Automatic segmentation of liver tumors
from MRI images
Rune Petter Sørlie
D EPARTMENT OF P HYSICS
U NIVERSITY OF O SLO
31 ST AUGUST 2005
Abstract
This thesis present three methods to segment liver tumors targeted to support cryo- or radiofrequencyablation treatment.
I have studied the characteristics of a liver image from an image processing point of view. My studies
show that the natural variations inside the liver will disturb the segmentation. Preprocessing experiments
were carried out to find what sort of filtering was needed. The speckle inside the liver were faded with
a median filter with a 5 × 5 filter kernel or with repeated use of a mean filter with a 3 × 3 filter kernel.
This preprocessing managed to reduce the effect of the speckle and preserve the turmor edges and thereby
provide the foundation for a successful segmentation. The segmentation methods I chose to implement
were watershed – for it’s wide range of applications and automatic features, level set method – for it’s
strength in segmenting volumes and the snake – which manages to conserve week edges. The first two
methods were implemented with use of the Insight Toolkit (ITK) open source library and the latter method
was implemented with Matlab from MathWorks. ITK was chosen for it’s wide range of medical image
processing functions. A snake implementation was available for Matlab and was chosen to save time.
The results show that all three methods may successfully segment a tumor provided the parameters are set
properly. The watershed method did not require an initialization while the other two required an initialization inside the tumor. If future work is aimed at an algorithm to automatically estimate the parameters, I
believe automatic segmentation can be used to support the therapy and thereby increase the quality of the
treatment.
Preface
This thesis reflects the work I have done during my Master’s Degree at University of Oslo. The project
was suggested by the Interventional Centre at Rikshospitalet University hospital. I was inspired by the
technology and the chance to make a small contribution to better patient treatment.
i
CONTENTS
CONTENTS
Contents
1
Introduction
1
2
Background
1
2.1
Medical imaging history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
2.2
Digitized imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
2.3
Pros and cons of MRI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2
2.4
Medical applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
3 Segmentation
5
3.1
Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
3.2
Thresholding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
3.3
Edge detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
3.4
Which dimension to use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
4 Methods
9
4.1
Snakes, an active contour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
4.2
Snakes steered with Gradient Vector Flow . . . . . . . . . . . . . . . . . . . . . . . . . .
10
4.3
Level set segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
4.3.1
Principle of the level set method . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
4.3.2
The level set primer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
4.3.3
The Fast Marching level set method . . . . . . . . . . . . . . . . . . . . . . . . .
15
4.3.4
The Threshold level set method . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
4.4
Watershed segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
4.5
Evaluation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
5 Implementations and results
21
5.1
Snakes with GVF-field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
5.2
Level set segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
5.3
Level set segmentation in 3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
5.4
Watershed segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
5.5
Evaluation of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
6 Discussion
35
6.1
Image features, the foundation for segmentation . . . . . . . . . . . . . . . . . . . . . . .
35
6.2
Promissing aspects for successful segmentation . . . . . . . . . . . . . . . . . . . . . . .
35
6.3
Aspects that complicates segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
6.4
Required preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
6.5
The methods evaluated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
6.6
Robustness of algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
6.7
Prefered method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
6.8
Preparing the segmentation output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
ii
CONTENTS
CONTENTS
7 Conclusion
41
7.1
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
7.2
Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
iii
LIST OF FIGURES
LIST OF TABLES
List of Figures
1
Relationship between intensity image, gradient map and gradient vector. . . . . . . . . . .
6
2
A GVF-snake iterated into concavity . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
3
Soap bubble with variable curvature. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
4
Movement of the curvature of the soap bubble. . . . . . . . . . . . . . . . . . . . . . . . .
12
5
Grass fire illustrates a propagating surface. . . . . . . . . . . . . . . . . . . . . . . . . . .
13
6
Model of two merging grass fires. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
7
Curvature in two dimensions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
8
Level set propagation term. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
9
Propagation term and upper threshold . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
10
Threshold level set method data flow chart . . . . . . . . . . . . . . . . . . . . . . . . . .
18
11
Watershed segmentation simplified to one dimension. . . . . . . . . . . . . . . . . . . . .
19
12
Liver tumor segmented with snake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
13
Snake segmentation failed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
14
Detail of challenging areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
15
Edge map of liver tumor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
16
Segmentation of tumor in multiple slices . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
17
Liver tumor segmented with Fast Marching level set segmentation . . . . . . . . . . . . .
26
18
Synthetic MR image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
19
The synthetic liver segmented in 3D with Threshold level set filter . . . . . . . . . . . . .
28
20
Synthetic tumor segmented in 3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
21
Tumor segmented from a real 3D MR image . . . . . . . . . . . . . . . . . . . . . . . . .
30
22
3D Segmentation of a large tumor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31
23
Watershed segmentation of liver tumor . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
List of Tables
1
Segmentation results of different level set algorithms of 2D MR-images. . . . . . . . . . .
26
2
Evaluation of the segmentation of the synthetic tumor. . . . . . . . . . . . . . . . . . . . .
29
3
Parameters used in watershed segmentation. . . . . . . . . . . . . . . . . . . . . . . . . .
32
4
Evaluation of segmentation results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
iv
2 BACKGROUND
1 Introduction
This work is a small and modest part of a quite complex system. The whole system will when completed
visualize the inside of the human body, and make surgeons able to perform operations inside a patient
without open surgery. Like laparoscopy, where needed tools are inserted into the body thru smaller holes.
In addition to the ability to see inside the patient we can provide additional information about the process
being done. For example when the treatment uses heating of tissue we can provide thermal information.
Of course the visualization is also very useful for navigating the instruments inside the patient.
More specifically the aim for this work is to segment a tumor in a liver. This will make the surgeon able to
see the tumor and then ease the treatment. The instruments needed for this could be ultrasound, computer
tomography (CT) and magnetic resonance imaging (MRI). An open MR system serves the image guided
surgery at The Interventional Centre. The open frame of the MR scanner combines the possibility to make
an image of parts of the patient interior and access to the same area. There is a time delay for each image
taken, however this is accepted for the increased image quality and precision.
2
Background
2.1 Medical imaging history
Nuclear magnetic resonance (NMR) was discovered by physicists in 1946. They found that the proton
spin orientation resonated for a certain magnetic field strengths. Different materials resonated at different
magnetic field strengths. The chemists were the first to make use of the discovery of magnetic resonance in
NMR spectroscopy. They studied physical and chemical properties of matter by the use of the NMR phenomenon. NMR spectroscopy make use of simple one-dimensional techniques to study chemical structures
[ritw].
The development of computers and numerical mathematics in the 1970’s made it possible to analyze NMR
in two dimensions. The very first image was of an orange and found place in 1972. The foundation of
magnetic resonance imaging (MRI) was born. In 1978 the first image of a human body was captured. Five
years later the MR imaging was approved for medical diagnostic by the Food and Drug Administration
(FDA) and we have been witnesses to a wide spread use since then [imaw]. Parallel with the development
of MRI did computer tomography (CT) evolve. CT was invented in 1972 and the first clinical CT scanners
were installed in the middle of the 1970’s. Back then hospitals were limited to ultrasound and x-ray pictures
for diagnostics imaging of patients .
Norway bought their first MR scanners in 1986. They were located at Stavanger and Trondheim. The
politicians thought this MR technique was some extremely expensive instruments bought to fancy hightech friendly doctors and surgeons. In a retrospective perspective it is easy to see that the first MR machines
were an important investment. Norway could build up our own knowledge in a new way to see inside the
human body. As with all new technique the prices fall with time and today there are about 72 MR scanners
for medical use installed in Norway.
MRI opens for new possibilities in a number of medical applications. As new modalities are developed it
opens for more specialization in existing modalities. The modalities work together peer to peer, like team
members in a cross disciplinary team. The same treatment of liver tumors done at Interventional Centre is
done with CT and ultrasound at other hospitals. There is no correct way to do it, but naturally focusing on
one modality and improving the belonging procedures will make a change.
2.2 Digitized imaging
We all recognize the x-ray film which are hung up on a light wall for inspection. The negative films are a
very convenient format. When they become larger the format is inconvenient for storing. It is also an film
expense for each picture which can not be neglected. The time had come to go digital and display medical
1
2.3 Pros and cons of MRI
2 BACKGROUND
images on computer displays. The images are stored in databases and fetched on demand by the medical
persons. The change to digitized images also opened for postprocessing the images. Postprocessing can
be adjustment of contrast and intensity or emphasizing a pattern or edges. Since the image is digitized the
natural tool for image processing is a computer program. The software based filtering opens for a new
world of possibilities. Those who analysis the images can apply filtering with many different properties.
Filter properties are often set by parameters which leads to unlimited number of ways to combine filters.
Trying many combinations would be very time consuming for the image analyst. So a smarter way would
be to develop a filter that does this job for the analyst. This means the filter has to adapt to the challenge it
is facing.
In scientific popular magazines we have seen nice color pictures of the inside of the human body. Colors
exist where it is light. The inside of the human body has no fancy colors unless it is lit up. The color
pictures are made on computers by applying different colors to each body part. Before the organs can be
tagged with a color it is necessary to decide which region belongs to which anatomical object. The process
of dividing the image into regions of organs is called segmentation.
2.3 Pros and cons of MRI
At first CT and MR images appear to provide similar information. While CT can depict soft tissue structures
much better than conventional x-ray, it does not reach up to MR images in contrast details. MRI can be
combined with the use of contrast agent which enhances organs and blood vessels in images. This makes
MRI superior to other modalities when it comes to analyzing soft tissue.
The principle of CT limit the images to trans axial slices. In comparison MRI can capture images in almost
any direction. The three basic directions are coronal, sagittal and axial orientation. It is also possible
to combine the three orientations for imaging almost any plane through the body. A MR-examination
will take more time than a similar CT-examination, typically 15 to 45 minutes. The MR-scanning speed
increased dramatically the first two decades and is still improving. In the early days real time images were
limited to ultrasound and flouroscopy. Real time images are useful for diagnosing body functions. For
example flouroscopy is used in gastroenterology1 and ultrasound used in cardiology2 . Today MR provides
real time images and is used in MR cardiology to diagnose the heart function. For a demonstration of real
time imaging a MR CINE, a short movie, of a beating heart is presented at University of California San
Diego’s web site [ucsdw]. The image quality is good and the image is easy to interpret compared to an
ultrasound image which requires a trained eye. The traditional MR-scanner required a cylindrical magnet
to provide enough field strength. The closed nature of the scanner will be a strain for patients suffering
from claustrophobic anxiety. Today there are alternatives in open MR-scanners. Both short bore MRmachines and C-shaped magnets provides patient access during examination at the cost of lower magnetic
field strength. MR and CT-scanners are stationary equipment where ultrasound and x-ray apparatus are
available as mobile units.
Both CT and x-ray imaging expose the patient for unwanted radiation and should not be used more than
necessary. In comparison, ultrasound and MRI examinations are considered harmless. The expense of the
equipment vary widely. Ultrasound apparatus and x-ray labs are available at lower cost, a fraction of the
cost of advanced MR scanner. But also here the gap is closing. Simpler MR systems are available at a cost
comparable to a full featured x-ray laboratory [Bell].
While MRI initially was mainly used to image the brain and spinal column MRI has developed to cover
a variety of body parts including joints, blood vessels, breast, as well as pelvic and abdominal organs like
the liver. The excellent contrast detail provided by MRI makes it very applicable for examining small
metastatic lesions (cancers) in the liver.
In addition MRI provides functional MRI (fMRI) which evaluates tissue function. For example fMRI assists a neurologist to locate which part of the brain that is activating a certain muscle operation by measuring
blood flow changes in the brain.
1 Gastroenterology: The medical specialty devoted to the study, diagnosis and treatment of disorders of the digestive system
[medw].
2 Cardiology: The study and treatment of heart disorders [medw].
2
2 BACKGROUND
2.4 Medical applications
2.4 Medical applications
The possibility to see inside the body opened for better diagnostic and surgery planning and improved the
understanding of body functions. Later medical imaging was also executed during surgery to assist and
monitoring the surgical procedures. As techniques and experience developed, open surgery changed to
minimal invasive surgery. For the abdomen area surgery through a small incision is called laparoscopy3 .
The laparoscopy is a very attractive alternative to open surgery because it stresses the patient far less. For
all surgery there is a risk of infections and by minimal invasive surgery the risk of infection is reduced
significantly. Another advantage is that the patient will restore to health sooner and will have less pain
when the surgery is performed with laparoscopy than with traditional open surgery.
One of the challenges with minimal invasive surgery is to provide the surgeon with enough information to
perform the surgical procedure in a safe way. When working with instruments through small openings the
surgeons want visual information, force feedback and even tactile feedback.
Several techniques are used for monitoring laparoscopy. We have optical systems which uses fiber optics
and lenses. Ultrasound is another widely used modality. Both methods allow easy access to the patient
while the image is retrieved. In the early days, MR and CT did not allow patient access while the part of
the patient which was of interest was inside the cylinder of the imaging machine. Later, the invention of
different open MR scanners offered a variety of new feasibilities to combine surgery and image capturing.
The Interventional Centre facilitates an Signa SP open MR system from GE Medical Systems. In stead of
the traditional tunnel the open MR machine is built with two large magnet rings. The image is captured
between these rings and therefore allows access to the area of interest while the image is captured. Since
MR images have many advantages compared to ultrasound the invention of the open MR scanner was an
important contribution to the evolution of laparoscopy.
We now have a general understanding of the value of monitoring the patient internals during minimal
invasive surgery. The surgical procedure this thesis support is ablation of liver tumors. There will be
diagnostic images from CT or MRI which provide information about both liver and tumor. The target of
the surgery is to defeat all cells in the tumor without destroying more than necessary of the surrounding
tissue. Since the liver is an essential organ the latter follows naturally. There are different ways to eliminate
tumor cells. Two of them are with freezing and heating. By freezing the tissue to about -40°C the cell
membranes are cracked and the cells killed. Another way is to heat up the cells until they are destroyed.
A heating method used frequently is the use of radio frequency (RF) alternating current. The technique is
called RF-ablation. Heat is generated locally by a probe with electrodes which can be shaped like prongs.
The electrodes are connected to an alternating voltage source causing alternating current to flow through
the tissue adjacent to the electrodes. The electromagnetic energy is converted to heat by ionic agitation,
which leads to coagulative necrosis4 inside a 3 to 5.5 cm sphere [medw, Hae01].
For both methods the precision of the probe placement will be essential for a successful result. There are
three steps where the imaging technology supports the process. First the planning is based on images. Then
the insertion of probes are done with guidance of an image system. Next the heating/freezing process is
monitored by the image system. Finally postoperative images are verifying if the surgery was successful.
So far hospitals use CT and/or ultrasound to guide cryo/RF-ablation. We hope to provide better results
with the open MR scanner. In addition to the patient access advantage of the open MR scanner there is an
advantage that MR images also provide some thermal information. 5 The thermal properties are modeled
by Samset [Sam01]. By combining information about the tumor and the thermal model we optimize the
ratio of probability that all tumor cells are defeated and amount of healthy tissue damaged. There will
be a temperature gradient from the center of the tumor out to normal body temperature in the surrounding
tissue. From this follows that a significant amount of fresh tissue will be frozen or heated to a level near total
damage. The dead cells are absorbed and digested in the bodys natural housecleaning process [ummw].
3 Laparoscopy – from Greek: lapara, "the soft parts of the body between the rib margins and hips", skopein, "to see or view or
examine." [medw]
4 Necrosis: The death of living cells or tissues.
5 The phase of the signal vary with temperature and provides thermal information.
3
2.4 Medical applications
2 BACKGROUND
4
3 SEGMENTATION
3 Segmentation
In image analysis and processing segmentation is an essential art. The ability to distinguish an object from
the background is wanted in machine vision on robots, in industrial applications like quality control, in
processing of seismic data and a number of other applications. I will use segmentation in a medical image
which in many ways are different from other images. Even if the sources of the images vary widely, many
techniques are general and will work in a variety of applications. In general we have a scene which we
capture in an image an then analysis the image. The scene is seldom optimal but rather a compromise of
several factors. For example could cost be an issue and time might also be limited. Further the sensor should
not interfere with the process monitored. The image is normally not perfect but has variable lightness and
noise. Other distortions may be introduced by the scene or the sensors. The good thing about this list of
problems is that general techniques are developed to reduce these difficulties.
3.1
Preprocessing
When I started studying MR images of livers the first impression was that the images suffered from a
significant amount of noise. After some further research on noise I found that the noise in MR images is
so low it can be neglected. The variations in the tissue which looks like random noise is mostly caused by
natural variations in the tissue. Normally we would reduce the noise by smoothing the image with a mean
filter. The drawback with filtering is that essential image details will disappear in the smoothing process.
The variations in the liver are natural variations in the tissue. Smoothing out tissue variations would remove
all image details that is needed for proper segmentation of the tumor.
The segmentation methods detects edges in the image. Edges vary between strong and weak depending
on how abrupt the change in intensity is. Without smoothing most methods will find edges everywhere.
This supports the concept of always using some smoothing of the image. To find the right balance between
smoothing and preservation of details is essential to achieve valid results in medical image processing.
Multiple segmentation methods worked with high degree of smoothing but details were lost. Another
problem is broken edges when the image is smoothed to much.
When using mean filter smoothing one should consider different filter kernels. Since the tumor has no
typical orientation, I can not see any reason why a non quadratic filter kernel should be best suited. I tested
5 × 3 and 3 × 1 filter kernels but the result was not good compared to using a 3 × 3 kernel. Details were
better conserved by iteratively smoothing with a small filter kernel than using a bigger kernel once. How
many iterations which were appropriate depend on the segmentation method we want to use.
Another problem for threshold based segmentation is variable background light. It is possible to estimate
the background variations and solve this problem by simply subtract the estimated background from the
input image. To calculate the background we use an smoothing operator large enough to cover the object
of interest. The background image is a smoothed version of the input image that is smoothed so much that
the image only contains slow variations. A typical background image can have darker corners or one side is
darker than the opposite. Correction for variable background light will in little degree influence on details
in the image. With a more homogeneous background it is less demanding to find proper parameters for the
segmentation method used.
Other challenges might be artifacts. In an image captured trans-axial of a major vessel, the vessel can
produce a repeated response in the image. Livers need plenty of blood to function and will always have
one or more vessel with large diameter. The movement of the blood in the vessel causes the MR machine
to misinterpret the signal as it came from another location. Therefore the image contains echo like spots
of the same size as the vessel. With advice from a radiologist I decided not to spend time on preprocessing
artifacts. Those images that had this artifact were considered mediocre MR images. There were better to
spend the time on other challenges. The same was the case for images that were blurry because of patient
movement during the scan. The radiologist meant if the image quality was low, we would rather run a new
scan than trying to refine the mediocre MR image.
5
3.2 Thresholding
3 SEGMENTATION
image
gradient map
GVF
Figure 1: Relationship between intensity image, gradient map and gradient vector.
Finally there is easier to segment a tumor that is located some distance away from other edges like vessels
or the edge of the liver. Methods for segmenting the liver is already described [Heu03] at previous work
at the Interventional Centre. That means I can assume the liver is segmented prior to my image processing
and it will not be too complicated to calculate how close we are to the liver edge. The vessels are not
segmented prior to my processing. How this issue was solved is described later.
3.2
Thresholding
In images that are not too corrupted by noise or distortion the human eye is superior to distinguish objects
from their background. Our sight is specialized in detecting edges. So how can we detect an edge? An
edge is made by an sudden change in color, intensity or texture. If the edge follows a line which have some
sort of regularity this strengthen the edge and opposite if the edge follows a random path it is harder to
follow the edge.
Transfering the human ability to detect edges to a computer program is a challenging task. The most
basic segmentation algorithm is the use of a global threshold for what is background and what is object of
interest. The objects will then be represented with regions. A drawback with a global thresholding method
is that it does not take into account what the neighbor pixels were set to.
3.3
Edge detection
Thresholding might be useful for many applications but provides no real edge detection. One approach to
find edges in images is detecting the steepest gradients. The gradient is found by making a differentiation
map of the image, also refered to as a gradient map. For a two-dimensional image there becomes two
maps, one for x-direction and one for y-direction. High values in the x-map indicates steep gradients in
x-direction, and visa versa. The two maps gives us both direction and magnitude of the gradient. For edge
detection the magnitude will be most valuable. Finding the magnitude is basic vector arithmetic where we
see the x-map as vectors pointing left-right and the y-map as vectors pointing up-down. The gradient in
each pixel is the sum of the x and y vector for the corresponding pixel. The magnitude is the length of the
gradient vector. The relationship between intensity image, gradient map and gradient vector is illustrated
in figure 1.
Edges in livers normally appear rather blurry. The higher gradient close to edges are spread over many
pixels indicating thick edges. We want a more precise decision of where edges are. One approach is to set
the edge where the gradient is at its highest. Finding the tops and bottoms are often done by differentiate
6
3 SEGMENTATION
3.4 Which dimension to use
again and then set the edge to the zero crossing. An algorithm to locate a zero crossing is simpler than
finding tops and bottom points.
When edges are found they need to be interpreted. Depending on the method used and image of hand we
might get anything from a perfect edge map which corresponds perfectly with the scene to a edge map full
of fragmented edges. The result depends strictly of how much smoothing you apply in the preprocessing.
3.4
Which dimension to use
MR scanners may produce three-dimensional (3D) output of the body under inspection. While the image
acquistion is done in 3D, the result is presented slice by slice so the 3D image is stored as a stack of 2D
images. Those 2D images are called slices and a volumetric image is called a slab. The fundamental
element in a 3D image is a voxel which is similar to a pixel in a 2D image. Each slice can typically be
a 256 × 256 pixel image and a scan of the liver might be for example 28 slices. It is possible to execute
the filter process on one slice at a time for all slices. The 2D approach will be simpler and less computer
intensive. A disadvantage with such a method is that it will not make use of neighborhood information in
the third direction. Working with 3D methods are more complicated than working in the 2D domain. To
increase the level of difficulty gradually I decided to implement and verify methods in 2D to get used to
different methods, their stronger and weaker sides, their parameters and robustness. Then at a later stage
3D methods were introduced and developed further.
The observant reader will notice that it is very different number of pixels in the third direction than in each
slice. From the meta information of the DICOM6 file we can find the resolution in all directions. The 3D
images I worked on had a pixel resolution of 1.5 × 1.5mm in each slice. Between the slices there are
3mm, which means half the resolution in the length direction (z-axis) than in x and y direction. Next the
scan only covers the part of the liver that is of interest. If images taken during diagnose shows the tumor is
located in a certain part of the liver the next scan can be limited to this part of the liver. The same does not
apply to each slice, since the way the MR scanner makes the image there will be no time-saving on limit
the image in x- and y-direction. In z-direction the time spent is proportional with number of slices made.
The size of the image is also proportional to number of slices and as the images are stored in databases
this is also an issue. Data storage are getting cheaper every year but cost is still an issue. A correctly sized
image will also imply faster transfers through the network.
6 DICOM:
Digital Imaging and Communications in Medicine. A standard for representation of images and reports.
7
3.4 Which dimension to use
3 SEGMENTATION
8
4 METHODS
4 Methods
Considering the huge amount of publications on medical imaging I expected to find several papers with
focus on segmenting liver tumors. An Internet search on the exact string “segmenting liver tumors” gave
only three hits. At first I found this quite remarkable. Lots of stuff was found on “liver tumors”. During
my work with the thesis I realized that one reason I did not find much stuff dedicated to segmenting liver
tumors was that segmentation is a general discipline. An awful lot of work is done on segmenting MR
images. By describing newly discovered algorithms in a generalized way they are aimed for more wide
spread use than if they were aimed at a narrower field like the one I work with.
During the theoretical part of the master’s education I learned that several different segmentation techniques could be used to come up with similar results. The methods typically varied in speed, accuracy and
robustness. Rather than trying to come up with yet another segmentation algorithm I wanted to select a few
promissing methods for comparison. In the following sections we will look into these three segmentation
methods:
• Snakes
• Level set
• Watershed
4.1
Snakes, an active contour
From the previous work on segmentation of the liver by Heuch [Heu03], we knew that segmentation with
snakes and gradient vector flow (GVF) field was successful. Segmenting a liver tumor is very different
from segmenting the liver but the strength of the snake algorithm was promissing.
Active contours are also described as snakes from about 1987 [Kas]. In contrast to segmenting by thresholding or edge detectors a contour will have an elasticity that is adjusted to the actual application. We can
think of an active contour as a snake that bend its body into the edge of the object. An important property
of an active contour is the ability to make continuous edges where the edges are weaker. Even if the edge is
weak or broken we can make a continuous contour. The continuous contour is made on the cost of details
in the edges. Smaller details will be smoothed out. Deeper concavities like U-formed objects will be a
problem as the contour will not be able to bend itself into the concavity. It will merely make a small dent
where the concavity is.
The idea with the snake is to let it act as an energy-minimizing spline. The energy in the spline depends on
its shape and its location in the image. The snake will not necessarily find a perfect contour in an image.
Rather the active contour will be steered by human interaction, or interaction with higher-level image of
some sort. The interactions could for instance come from an edgemap of the image.
The contour is described as a parametric curvature v(s) = [x(s), y(s)] in a cartesian xy-coordinate system
where s is a parameter in the interval [0, 1]. Usually v(s) is a spline. A snake is a special case of a more
general technique to make a model fit into an object with use of energy minimization [Son]. The energy
equation is given by the integral
Z
∗
Esnake
=
1
{Eint (v(s)) + Eimage (v(s)) + Econ (v(s))}ds
(1)
0
Where Eint is the energy in the spline, Eimage comes from forces in the image and Econ from external
constraint forces. The last two are energy from outside forces. The internal energy can be described with
a differential equation where the first and second differentiate of the curve v describes stretch and stiffness
in the spline. If we use the analogy with the snake it is expandible like a rubber band and works against
bending. The properties can be adjusted with the parameters α and β in equation (2).
9
4.2 Snakes steered with Gradient Vector Flow
Z
Eint =
0
1
4 METHODS
1
(α|v 0 (s)|2 + β|v 00 (s)|2 )
2
(2)
The external energy will typically come from image edges. The edge energy are calculated with a gradient
operator. Sharp and abrupt edges results in higher gradients. The blurred out and weaker edges gives lower
gradient values. It is also possible to use image intensity to steer the snake towards lighter or darker areas.
Another approach is to let the snake attracts to terminated lines. This will be useful for images where the
edge is missing but a virtual edge is defined by the end of lines. If we stay with the gradient approach we
get
Eimage = −|∇I(x, y)|2
(3)
Here I(x, y) is the image function. The minus sign is to get minimum energy where the edge is strongest.
Figure 1 on page 6 can be used as an example where the left image is I(x, y). The gradient map will then
represent the energy from image forces, |∇I(x, y)|.
4.2 Snakes steered with Gradient Vector Flow
As already mentioned the snake need an initialization. If the energy is controlled only by gradient the
initialization need to be quite close to the edge. Another limitation is the snakes missing ability to bend
into concavities.
Both these limitations are solved with the introduction of gradient vector flow field as external forces as
described by Xu and Prince [Xu97]. In stead of using an energy minimizing function they make use of an
force balance equation. To minimize energy the snake must satisfy the Euler equation, given in equation 4.
Note the change in notation. Now each pixel is given by vector x rather than xy-coordinates as above.
αx00 (s) − βx0000 (s) − ∇Eext = 0
(4)
which can be written as a force balance equation
(p)
Fint + Fext = 0
(5)
(p)
where Fint = αx00 (s) − βx0000 (s) and Fext = −∇Eext
To solve these equations we let the snake-function x be a function of both s and time, x(s, t). To find the
snake position for equilibrium we set the partial derivative of x, xt (s, t) = ∂x/∂t to zero. Using this in
equation 5 we get
xt (s, t) = αx00 (s) − βx0000 (s) − ∇Eext
(6)
The equation is solved when x(s, t) stabilizes causing xt (s, t) to vanish. A numerical solution is found by
discretizing the equation and solve it iteratively.
Where the gradient is zero in the homogeneous parts of the image there will be no external energy to
force the snake towards the edges. This is solved by introducing a computational diffusion process. After
diffusion there will be no homogeneous areas in the image, only various degrees of gradient. The result
of the diffusion process looks much like a smoothed version of the input image. Where the image was
homogeneous there now is a small gradient toward an edge.
To make a GVF field that is able to force the snake toward an edge Xu uses a trick. He is setting the gradient
to unity magnitude and keeping the direction. At this point an illustration will better describe the process.
Figure 2 on the facing page shows how the snake manages to find its way all the way into the concavity.
For simplicity a binary image is used for illustration. The U-shaped object would cause problems for
the traditionally parametric snake. In the first image the rings around the object shows the initial snake
and movement for each fifth iteration. The fourth image show the GVF-field after a successful diffusion
10
4 METHODS
4.3 Level set segmentation
Deformation, iter=125
(a)
edge map
(b)
edge map gradient
normalized GVF field
(c)
(d)
Figure 2: A GVF-snake iterated into concavities of a U-shaped object (a). Detail of concavity enlarged by
a factor of three (b),(c) and (d).
process. The snake converges to the object edges. We see that coarse initialization is not a problem as the
snake reaches the object after about 20 iterations. The next challenge is to detect the concavity. The image
shows that further iterations brings the snake into the concavity before it converges at the bottom. This
seemed so promissing that I decided to try GVF-snakes on liver tumors. This work is presented in a later
section.
4.3
Level set segmentation
The level set methods are natural to consider using for many problems in three dimensions or even higher
dimensions. What ever dimension the data are describing level set methods solve the problem by adding
one dimension to the problem. Adding a dimension will at first glance seem to complicate the process but
as we will see later this is not the case. Segmenting images with level set methods was introduced at the
end of the 1980’s and was based on previous work on moving curvatures. Since then several variants and
improvements have come up. Some of the improvements are aimed at speeding up the processing. Other
methods have strength related to specific challenges like noise and broken edges.
4.3.1
Principle of the level set method
The edges I want to track in my images are what they call boundaries or interfaces in the level set literature.
There boundaries are divided in dynamic and static boundaries. Examples of the first are an ocean wave
or a developing fire. An tumor in a medical image is an example of the latter. Often the properties of an
boundary can be described by a physical or chemical process. Even if it is possible to make a model of the
problem of hand, it might be very complicated to forecast a moving boundary. How we decide to describe
the boundary mathematically will be essential for how the problem of tracking a moving interface is solved.
One way could be to present the interface in a coordinate system. For higher flexibility a parameterized
cartesian description is often chosen. Unfortunately parameterizing has limitations when it comes to sharp
corners or a change in topology like when two interfaces meet and form a new one.
Lets illustrate the problems mentioned with a few examples. For simplicity we first limit the problem to be
two-dimensional. A soap bubble is blown off the dispenser. We look at the boundary in a plane through
the center. Just after leaving the dispenser the side against the blower is pressed in by the blowing air like
illustrated in the figure 3 on the next page.
11
4.3 Level set segmentation
4 METHODS
Figure 3: Soap bubble just leaving the dispenser.
Figure 4: Movement of the curvature of the soap bubble.
The boundary of the bubble will move with a speed proportional to the surface tension which again is
proportional to the curvature. We know from experience that the bubble very quickly turns into a perfect
sphere or a circle in our example. So how would we describe this problem if we should calculate the
moving interface on a computer. If we parameterized the curve by walking around it and drop points at a
certain rate we would have a fine numerical description of the boundary as show in figure 4.
This solution seems fine at first but let us complicate it a bit. We blow abrupt at the bubble so it splits
to two smaller bubbles. If we wanted to study the process of the fission in detail we were running into
trouble. How should we know which points are connected together by the surfaces? We might find a
way to track this in two dimensions but if we move over to three dimensions the problem might be more
than we can handle. A topological change caused by a merge of two propagating fronts would be even
more complicated to handle. In addition to keeping track of which points should be connected, superfluous
points should be sorted out and removed.
To get around these problems Sethian introduced the level set approach for describing the problem of a
moving interface. His trick is to describe the problem in a coordinate system but add an extra dimension
to the problem. Instead of tracking a number of points around the curve as the interface is moving we start
with an initial curve in xy-plane and make new ones in the z-plane. The change in z-direction is defined by
the z-function z = φ(x, y, t) where z = φ(x, y, t = 0) is the initial curve. This z-function is the level set
function. The level set function is a surface in our three-dimensional room describing the two-dimensional
process under investigation. Sethian is using a spreading grass fire as an example of a moving interface. If
the surface is homogeneous and there is no wind we can assume a constant propagation speed of the front.
The level set representation of the moving fire will form a nice cone as illustrated in figure 5 on the next
page.
To find out where the interface of the fire is at a certain time you can just find the intersection between
the surface and a parallel-plane to the xy-plane elevated to the z-level representing the given time. The
real gain with this method is the handling of meeting interfaces. While this caused some problems in a
12
4 METHODS
4.3 Level set segmentation
12
10
z
8
6
4
1
2
0
0
−1
−1
−0.5
0
0.5
1
y
x
Figure 5: Illustrates the moving front of a spreading grass fire. The z-axis represent time.
parameterized representation the level set presentation handle the topological change very elegantly. The
boundary is still the intersection between one or more surfaces and the plane representing the time. When
two surfaces meet the resulting surface is simply the outside of the two joined structures. Sethian claims the
topological problem vanishes. You do not have to deal with the motion of the front. The implementation
just need to check if you are outside both interfaces, you are outside. If you are inside one or more of the
interfaces, you are inside. This is illustrated in figure 6 on the following page where a fire spreading from
a point is merged with a distant fire moving as a linear front with the same propagation speed.
4.3.2
The level set primer
First I will describe the level set method in detail. In the next section I will give an overview of the principle
of a speeded up algorithm called Fast Marching Level Set Method. Finally the Threshold Level Set method
is presented in a separate section.
Start with a simple smooth closed curve γ(0) in R2 . Then γ is a family of curves where the parameter t
represents the time in the range from zero to infinity. The curves are based on the initial one which moves
outwards. The movement is normal to the tangent and with speed F . The speed function F is a function of
the curvature K.
In the plane the curve γ(t) is parameterized with s by the position vector X so X(s, t). There s varies from
zero to S and X(0, t) = X(S, t) ensures a closed curve. When s is increasing we move in positive angel
direction (counter clockwise). In the same way we parameterize the curvature to K(s, t).
Now, lets take a closer look at curvatures. For a parameterized curve the curvature κ is defined as
κ=
dθ/dt
dθ/dt
dθ
=
=p
ds
ds/dt
(dx/dt)2 + (dy/dt)2
(7)
p
dx2 + dy 2
(8)
ds =
The relation between ∆s and ∆θ is illustrated in figure 7 on the next page.
By expressing tan θ and dθ/dt with the derivatives x0 = dx/dt and y 0 = dy/dt, combine the equations
and perform a couple of small tricks we finally get
13
4.3 Level set segmentation
4 METHODS
z
10
5
0
1
0.5
0
−0.5
y
−1
0
−0.5
x
−1
0.5
1
Figure 6: Model of two merging grass fires. A fire started at a point meeting a linear front and forming a
new merged interface.
Figure 7: Curvature in two dimensions. Moving a distance ∆s changes the tangent ∆θ.
κ=
x0 y 00 − y 0 x00
(x02 + y 02 )3/2
(9)
as curvature function expressed in the derivatives of x and y with respect to t [wolf05]. The curvature
for a two-dimensional curve given implicitly by g(x, y) = 0 [Gray97]. Gray use the notation where the
subscript of g indicates it is a partial derivative. 7
κ=
gxx gy2 − 2gxy gx gy + gyy gx2
(gx2 + gy2 )3/2
(10)
The theory presented so far are inspired by Osher’s and Sethian’s work [Osh88]. Osher goes further with
proofs for those interested.
Let us now study the connecting link between the resulting Hamilton-Jacobi equation and a hyperbolic
conservation law. Recall that the moving front has curvatures γ(s, t). Then we need a function describing
the propagating speed of the interface.
We have a closed simple connected curve meaning it is not crossing itself. Then we let the level set surface
φ move so the zero level set at any time is representing the moving front γt . By studying a point x(t) on the
curve of a level set moving φ(x, t) = C which means φ(x(t), t), we find the point is moving with speed
7 The
notation is gx =
∂g
,
∂x
gxx =
∂ ∂g
∂x ∂x
=
∂2 g
,
∂x2
gxy =
∂ ∂g
∂y ∂x
14
=
∂2 g
∂y∂x
4 METHODS
4.3 Level set segmentation
∂~x/∂t. The direction ~n is normal to the level set tangent at x. The speed is given by the speed function F
[Ada95] so we can write
∂~x
· ~n = F
∂t
(11)
−∇φ
|∇φ|
(12)
n=
∂~x
∇φ = F |∇φ|
∂t
(13)
By the chain rule and replacing the second part with equation (11) we get
φt +
∂~x
∇φ = 0
∂t
φt + F (K)|∇φ| = 0
(14)
(15)
This is a differential equation where φ is a solution and at any time the curvature γt is the zero level set of
the level set function φ.
Recall Grays formula for curvature of a two-dimensional curve equation (10). With a curvature dependent
speed function we can express the F (K) function by the partial derivative of φ.
F =
φxx φ2y − 2φxy φx φy + φyy φ2x
(φ2x + φ2y )3/2
(16)
Here we made use of a fixed coordinate system related to the physical domain. This is called a Eulerian
formulation for a propagating interface. The advantages of using such a method is first that fixed underlying coordinate system avoids instability known to be an issue in Lagrangian approximations. 8 Further
topological changes caused by merging or splitting of surfaces are handled implicitly. And finally the issue
that makes me so interested in the topic, the problem of tracing an interface can be extended to find a
moving surface in a three-dimensional data set. Unlike the 2D curvature function given in equation 9 and
10 the curvature of a 3D surface is described with mean or Gaussian curvature.
4.3.3
The Fast Marching level set method
In the previous section we introduced the level set method and it’s advantages over alternative methods.
Now we will set focus on the computational labor performed by a level set algorithm. Since the level set
methods solve the problem in one dimension higher than the problem domain, number of operations also
increases. For a two-dimensional problem described by n × n discretized points the number of operations
performed for each time step becomes n3 . As dimension increases the computational labor increases
dramatically because of a complete recalculation of the speed function for each time step.
The solution to this heavy computation is to not calculate the full speed function but limit the computation
to include points close to the propagating front [Ada95, Set95]. In the literature this is called the narrow
band approach. To use the fast marching level set method the interface must advance monotonically. The
front can expand or extract as long as the hole curve moves outwards or inwards.
The principle of the narrow band approach build a tube rather than the hole level set function φ. The tube
represent the full level set surface for a narrow band outside or inside the initial curvature depending on
the moving direction. The tube include all points with a distance less than a given maximum distance from
8I
have not covered Lagrangian approximations.
15
4.3 Level set segmentation
4 METHODS
the initial curve. A distance function is used for these calculations. If the narrow band span over k grid
points the number of operations is reduced from n3 to n2 k. 9 Further computational savings are won by
not calculate a new tube at every time step but keep the tube as long as we can without risking the curvature
overruns the edge. Instability is prevented by an algorithm that detects and avoids the tube edge being
crossed. Given an initial curve supposed to expand, the algorithms steps will be:
1. Build a tube that defines the φ function.
- Tube include all points with distance d < dmax from the curve.
2. Calculate the speed term for k grid points out from initial curve.
3. Move front according to speed term.
4. If interface is within tube goto 2) to iterate.
5. Go to 1) to reinitialize the tube with the current curve.
If the curve was initialized outside, the algorithm moves k grid points in from initial curve in step 2. Then
the matrix used for computation could be redused in next reinitialization to save memory and computational
time. The essence of the algorithm is to provide the calculations of the level set function with the necessary
data for each iteration. Those computations that will not be used until later iterations are postponed.
4.3.4 The Threshold level set method
The threshold level set method is based on the region growing technique, connected-components. The idea
of using thresholds is motivated by the assumption that different tissues have different intensities in MR
images. By defining a range of intensities that is typical for the tissue of interest we use the thresholds to
control the propagation of the moving surface. By combining the connected-components technique and a
level set method we get an algorithm that is more resistant to leaking through weaker edges.
We define the propagation term of the differential equation of the level set function being P . The input
image is given by a g function. Then the propagation is defined as
½
P (x) =
g(x) − L , g(x) ≤ C
U − g(x) , g(x) > C
where C =
(17)
U −L
+L
2
The result of propagation term P is a function that provides a suitable speed map for the propagating
surface. How threshold values affect the speed term is illustrated in figure 8. The variation in intensity
along a horizontal line through the tumor is plotted as a profile. The vertical gridlines matches where the
profile intersects the upper threshold. A positive speed leads to a expanding model and negative speeds
cause the model to contract. To get a positive propagation term everywhere inside the tumor the lower
threshold must be set properly. If the lower threshold is set to L = 110 we get P (x) = 0 inside the tumor
as well as along the edges. Zeros inside the tumor will lead to unsegmented spots which could be removed
by postprocessing. Spots inside the tumor is not necessarily critical but a certain propagation speed inside
the tumor ensures faster convergence.
By varying the upper threshold we can control how wide the surface becomes before it stops propagating.
The upper threshold’s influence on the propagation term is illustrated in figure 9 on the facing page. The
figure shows how the zero crossings are spread with increasing upper threshold.
9 Sethian
says the number of operations reduces to nk2 which I think it is a typing error. Adalsteinsson [Ada95] says n2 k.
16
4 METHODS
4.3 Level set segmentation
(a) MR image of a liver tumor
180
150
P(x) and g(x)
120
90
60
70
30
0
110
0
7
29
pixel position (x)
(b) Intensity g(x) versus speed term P .
Figure 8: Propagation term and lower threshold. The top graph shows the intensity profile in a line through
a liver tumor. The lower graphs show P terms with upper threshold set to U=150 and lower threshold L
varies between 70-110.
160
P(x) and g(x)
140
U=160
0
U=140
0
5
8.5
27
31
pixel position (x)
Figure 9: Propagation term and upper threshold. The top graph shows the intensity profile in a line through
a liver tumor. The lower graphs show P terms with lower threshold set to L=90 and upper threshold set to
U={140,150,160} iteratively.
17
4.3 Level set segmentation
4 METHODS
Figure 10: Data flow for the threshold level set method applied to a segmentation task. Courtesy Insight
Software Consortium, http://www.itk.org.
The threshold level set filter requires a feature image and an initialization image as input. All parameters
needed are the upper and lower thresholds. An example of a complete image processing pipeline is illustrated in figure 10. Here the initialization image is created by a single seed point and a distance processed
with the fast marching filter. Upper and lower threshold are internal parameters and not shown. A weighting of the curvature term versus the propagation term is done with two parameters. The segmented output
from the level set filter is sent through a binary threshold filter to provide a binary image.
18
4 METHODS
4.4 Watershed segmentation
Figure 11: Watershed segmentation simplified to one dimension.
4.4 Watershed segmentation
Watershed segmentation uses the analogy from topography. By interpreting the gradient map of an intensity
image as height values we get lines which appear to be ridges. If the contours were a terrain, falling rain
would find the way from the dividing lines towards the connected catchment basin. These deviding lines
are called watersheds. Figure 11 illustrates that steep edges cause high gradients which are watersheds.
The edge feature map is not limited to gradient magnitude. Other measures such as texture and intensity
can be combined to get the best possible separation of the classes. The homogeneous areas of the image
cause low edge feature values and forms the catchment basins. There might be an idea to truncate values
below a given threshold to reduce the number of regions detected.
The initial development of watershed segmentation where meant for topological digital elevation models.
The algorithm used a 3 × 3 neighborhood to detect potential watersheds. The results were connected in a
geomorphological network. Even if the principle of the watersheds are easy to understand by its analogy to
a terrain model it was difficult to find a robust algorithm. The first algorithm was both slow and inaccurate.
Parallel to the initial development there were investigations on watershed segmentation in digital images as
well. There were two approaches. The first one finds the downstream path for each pixel to a local altitude
minimum. The set of pixels that lead to the same minimum defines a catchment basin. This algorithm
is also known as a top-down approach. The approach was both computer intensive and inaccurate. The
second approach was presented by Vincent and Soille [Vin91]. Their algorithm proved more practical by
just change the way the catchment basins were filled. They thought of the gradient map as a terrain model
where a small hole were applied to each local minimum. Then the model was immersed in water and the
catchment basins started filling. Therefore this algorithm is called the bottom-up approach. As the basins
fills up some of them will eventually merge. To prevent merging imagine a dam is built from where water
is flooding over to a neighbor basin towards the higher surroundings. This way the watersheds form as the
surface model is immersed.
Vincent and Soilles algorithm sorts all pixels in the gradient map in increasing gray level. During sorting a
histogram is created and a list of pointers to all pixels with given gray level. Next step is the flooding fast
breadth-first scanning of all pixels gray level. When flooding has come to level k, all pixels with values
lower than k are already assigned a basin label. Pixels with level k + 1 is the next to be processed. Pointers
to these pixels are in the sorted list and therefore possible to access directly. The direct access speeds up
the algorithm significantly. If a pixel with level k + 1 is a neighbor of a labeled basin it belongs to that
basin. Pixels that does not have a neighboring connection with existing labeled basins defines a new basin
with a new unique label.
The watershed segmentation algorithms produces a region for each local minimum. This will normally lead
to over segmentation. We can say the algorithm has solved the problem but delivers the result as a puzzle.
Obviously there is a need for postprocessing these numerous regions. One way to face this problem is to
19
4.5 Evaluation methods
4 METHODS
organize the regions in a hierarchy. To get a measure to evaluate if a region should be separate we can use
the depth of the catchment basin. If the depth is below a certain value we continue to fill up so the region
merges with a neighbor region. Another measure could be the region size. These measures and others can
be combined in a way which suits the application on hand.
Previous work also suggest the postprocessing to be done manually. In an image of organs knowledge of
the anatomical structure is essential. By using graphical user interface (GUI) a qualified technician can
quickly connect the regions belonging to the organ of interest.
4.5
Evaluation methods
In the previous sections I have presented a few algorithms which seem promissing for tumor segmentation.
The next step is to evaluate the performance of the various methods. The need for a measure of the quality
of a segmentation is not specific to tumor segmentation, rather it is of general interest in segmentation of
medical images. The performance of a segmentation will influence on both the ability to detect and monitor
a disease. Thus, preoperative planning and operative surgical procedures requires segmentation methods
with high reproducibility.
To perform an evaluation of a segmentation it is necessary to verify against a known pixel- or voxelwise gold standard [Zou04]. This report search to classify two classes, healthy tissue and tumor tissue.
Unfortunately it is impractical to decide the true tissue type of each voxel in a MR image. Alternative
methods are developed to overcome the problem of the unknown truth. Physical phantoms with a known
truth of spatial location of tissue classes are developed. However, it is difficult to simulate the human
body’s response to magnetic resonance. Therefore the radiology community strive to develop realistic
and assessable models for research and training. Anyway, the phantoms available do not reach up to a
real medical image with all it’s properties as partial volume artifacts, intensity heterogeneity, noise and
anatomic variability.
The Dice Similarity Coefficient (DSC) is often used to measure the accuracy of an automatic segmentation
algorithm. The method is simple and appraise the spatial overlap of two binary images. The coefficient
ranges from 0 – for no overlap, to 1 – for a complete overlap. Dice similarity coefficient expresses the
overlap of two regions relative to the sum of the two areas. For two regions A and B, the DSC is given by
equation 18.
DSC(A, B) =
2(A ∩ B)
A+B
(18)
A second aspect to consider is robustness of algorithms with regards to noise, weak edges and artifacts.
Unlike the DSC, I did not find a general measure for robustness of an algorithm. However, all experiments
were logged with parameters, instability and a subjective evaluation of the segmentation result.
Finally, there is often of interest how fast the different algorithms are. Actually, speed was the least important property of the algorithms according to specifications from the Interventional Centre. Still, it was
practical to take notes on the time spent by each segmentation algorithm. This was simply done by using
the GNU Linux command time and then take notes.
20
5 IMPLEMENTATIONS AND RESULTS
Deformation finished, iter = 20
y−coordinates (pixels)
y−coordinates (pixels)
Estimated edge of the tumor
10
20
30
40
10
20
30
40
10
20
30
40
x−coordinates (pixels)
10
20
30
40
x−coordinates (pixels)
Figure 12: Successful segmentation with snake. Left image shows MR image and the segmented edge.
Right image shows GVF field, initial snake and iterations.
5 Implementations and results
To verify the theories of segmentation methods I tested the algorithms ability to segment a liver tumor in
MR images. I was provided with a number of two-dimensional MR images. The three-dimensional MR
examinations were more scarce and only two were provided. Normally diagnostic imaging of patients with
cancer is done with computer tomography (CT). Since the possible future use of the algorithm is to support
surgery in the open MR scanner I limited the testing to MR images. When an algorithm required multiple
parameters to be set I was facing an optimization problem. To get more control of how intensity variations
affected the algorithms I developed a synthetic MR image of a liver with a tumor.
5.1 Snakes with GVF-field
From previous work [Heu] we know the need for proper initialization of the snake for valid segmentation
result. In difference from segmentation of the liver which is done preoperatively, I would search for a
simpler initialization. Unlike lung tumors which has a complicated shape, a liver tumors has a more
regular shape. Based on the empirical knowledge of the shape I decided a circle would be a good general
initialization. The circle would require two points to be marked. This could easily be done in a GUI
were the center of the circle and the radius were given with two clicks. After discussing the user interface
issues with experienced medical personal I realized a one-click initialization was strongly prefered over a
two-click.
Another issue is whether to initialize the circle inside or outside the tumor. Starting the contour around the
tumor works well when there is enough space between the tumor and the edge of the liver. Else the snake
will be attracted to the liver edge rather than the tumor. This problem could be solved since information
about the liver edges are available. In stead of introduce a complicated liver edge remover I considered the
alternative initialization. Another issue is blood vessels inside the liver. By initializing the snake from the
inside of the tumor it will not meet other edges than it’s own.
After studying lots of MR images of liver tumors I concluded that those treated were always of a certain
size. A typical liver tumor was 2–3 cm in diameter and sometimes much bigger. I never found an example
of a tumor smaller than 2 cm that was planned treated. This led me to the idea of reducing the manual
21
5.2 Level set segmentation
5 IMPLEMENTATIONS AND RESULTS
initialization to specify a point near the mass center of the tumor. Then I made a circle with diameter about
2 cm and chose 12 points on the circle as the initial points of the snake. A smaller circle will work well
as long as the most dense part of the tumor is inside the circle. Then the GVF field will point outwards
from the center of the circle. Making the circle too small will increase the probability that the snake will
not expand in all directions. For close initializations the circle might be partly inside and partly outside of
the tumor. Then the snake will partly expand and partly contract. This is the perfect initialization which
ensures a fast convergence of the snake. The 2 cm initialization will also work for smaller tumors as long
as the tumors are not to close to other edges.
Implementing the GVF snake in Matlab was quite easy with Xu’s GVF toolbox [xuw] installed. Xu provides functions for calculation of the GVF field, snake initialization and snake deformation. To ensure a
useful GVF field some preprocessing to smooth out the liver texture was necessary. A mean filter could
be used but the median filter was chosen for its better edge preserving features. Using a 5 × 5 filter kernel
gave the best edge preservation but marginally enough smoothing. Figure 12 on the preceding page shows
the smoothed MR image of a liver tumor. The corresponding GVF field with the initial snake and iterations. Finally the converged snake is plotted in white on the MR image to verify that the segmentation was
successful. 10
For evaluation of the robustness of the snake algorithm I did several different initializations. A potential
problem was discovered as seen in figure 13 on the next page. When initialized with a small circle the
snake fails to deform out to the lower left part of the tumor. In figure 14 on the facing page the area of
interest is magnified. We see at pixel (17, 27) that a couple of vectors in the GVF field is pointing opposite
direction of the main stream. The vectors caused the snake to converge to early. Here the parameters that
controls the behavior of the snake comes to focus. The snake proved to be little sensitive to parameter
settings. I did not manage to break trough the two vectors with adjusting the parameters. The only way to
solve the problem was to increase smoothing until the vectors turned around. The smoothing also washed
out details and was not a good solution.
It was of great interest to find out if the snake algorithm would be useful for 3D MR images. When we want
to segment multiple slices it is natural to start with a slice that is near the center of the tumor. I wanted to
find out if the result of the first segmentation could be used as initialization in the neighbor slice. As long as
the tumor had about the same size and location in the xy-plane this initialization procedure worked well. If
the tumor decreased in area from slice to slice the previous segmentation became an outside initialization.
The outside initialization was still a challenge and caused the snake to attract to edges in the liver tissue
rather than the tumor. The problem was partly solved by reducing the size of the initial snake. By using
a circle one third of the area of last segmentation I usually hit a valid initialization. The problem was
now that the center of the tumor was not necessarily at the same point in the xy-plane from slice to slice.
I considered taking hight for this by estimating the line through the tumor center in each slice. This is
illustrated with a simple drawing in figure 16 on page 24. There were room for improvements, however
this was what I managed to implement.
5.2 Level set segmentation
Methods for handling level set segmentation are provided by the Insight Toolkit (ITK) [itkw] by the National
Library of Medicine (NLM). This is an open source C++ library under active development. It provides lots
10 Neither the monitor nor the printer were calibrated for gray levels so the tumor might appear smaller than the final segmentation
result.
22
5 IMPLEMENTATIONS AND RESULTS
5.2 Level set segmentation
Deformation finished, iter = 20
y−coordinates (pixels)
y−coordinates (pixels)
Estimated edge of the tumor
10
20
30
40
10
20
30
40
10
20
30
40
x−coordinates (pixels)
10
20
30
40
x−coordinates (pixels)
Figure 13: Challenging part of the segmentation. Right image shows alternative initialization leading to an
unsatisfying segmentation result (marked with a box).
Estimated edge of the tumor
Deformation finished, iter = 20
22
y−coordinates (pixels)
y−coordinates (pixels)
22
24
26
28
30
24
26
28
30
32
32
12 14 16 18 20
x−coordinates (pixels)
10
22
15
20
x−coordinates (pixels)
Figure 14: Zoomed into the challenging area. Notise the two vectors near pixel (17,27). They point against
the main stream and causes the snake to stop there.
23
5.2 Level set segmentation
5 IMPLEMENTATIONS AND RESULTS
Edge map
Histogram equalized edge map
Figure 15: Edge map used for input to calculate the GVF-field. In the right image the edge map is histogram
equalized to emphasize how rich of edges the liver is.
Figure 16: Segmentation of tumor in multiple slices. The center of mass in each slice is close to the same
line.
24
5 IMPLEMENTATIONS AND RESULTS
5.2 Level set segmentation
of classes and functions useful for segmenting medical images. And the best of all, it does so in two, three
or even higher dimensions.
To keep the complexity low I started with segmenting a 2D liver tumor. I had to study the ITK-library
and documentation for a while to overcome the user threshold. When the principle they were using were
understood it was quite easy to set up a chain of filters. All the time the data is hidden in an object which
efficiently encapsulates the data and thereby ensures the integrity of the image. The encapsulation used by
the ITK-library required a different way of working compared to Matlab were I at any time can visualize the
result of a filter process by a single command. The ITK-library does not provide any visualization functionality. The simplest way to visualize the result is to store the data to a file and view the file with a suitable
program, for instance Matlab. For a seamless solution where visualization is required The Visualization
Toolkit (VTK) is recommended. Visit their web site [vtkw] for the details.
The filter chain mentioned above is an essential part of an itk-segmentation program. You start with a file
reader filter and connects it with the preprocessing filter of choice. Further, the output from the preprocessor
is connected to the segmentation filter. Finally the output of the segmentation can be connected to a
postprocessing filter or directly to a file writer filter. To activate the filter chain on the data we simply
send a update message to the file reader. We then send a new update to the file writer filter. The message
walks backwards through the filter chain. To catch eventual errors these essential lines of code are put in a
try/catch statement as shown by the code below.
try
{
reader->Update();
fastMarching->SetOutputSize(
reader->GetOutput()->GetBufferedRegion().GetSize() );
writer->Update();
}
catch( itk::ExceptionObject & excep )
{
std::cerr << "Exception caught !" << std::endl;
std::cerr << excep << std::endl;
}
When used with 2D images the segmentation was done in a few seconds. Figure 17 on the following page
shows the result from a fast marching level set segmentation of a liver tumor. The left image shows the
hole liver and the right shows a magnified image of the tumor.
The other level set algorithms provided by ITK were also tested. The full list is
- Fast Marching Segmentation
- Shape Detection Segmentation
- Geodesic Active Contours Segmentation
- Threshold Level Set Segmentation
- Canny-Edge Level Set Segmentation
- Laplacian Level Set Segmentation
The results of these segmentations are listed in table 1 on the next page. Then we proceed with level set
methods in three dimensions.
25
5 IMPLEMENTATIONS AND RESULTS
y−coordinates (pixels)
y−coordinates (pixels)
5.2 Level set segmentation
120
140
160
180
200
100 120 140 160 180
x−coordinates (pixels)
130
140
150
130 140 150
x−coordinates (pixels)
Figure 17: Liver tumor segmented with Fast Marching level set segmentation. Initialized to (135, 135), σ =
1.0, α = −0.20, β = 4.0, threshold = 100, stopvalue = 100. Left image covers the liver. Detail of
tumor in right image.
Level set method
Fast Marching
Shape Detection
Geodesic Active Contours
Threshold
Canny-Edge
Laplacian
result
good segmentation
good segmentation
satisfying
good segmentation
satisfying
lack of details
Table 1: Segmentation results of different level set algorithms of 2D MR-images.
26
5 IMPLEMENTATIONS AND RESULTS
5.3 Level set segmentation in 3D
120
80
130
100
140
120
150
140
160
160
180
170
80 100 120 140 160 180
110 120 130 140 150
Figure 18: Synthetic MR image. Slice 35 of the synthetic liver with a tumor at (128, 150). Images magnified by a factor of 2 and 3. Mean intensities of liver and tumor are 80 and 60 of a maximum of 255.
Variance of Gaussian noise 0.001. Tumor is a sphere with radius 12.
5.3 Level set segmentation in 3D
The ITK-library is written to be able to handle 3D images. This means I could build the code for 3D
segmentation on my previous code. The variables describing the data model needed to be adjusted for three
dimensions. Initialization input needed to handle a point in (x, y, z). Else there was no big differences in
the code. On the other hand the new output was very different from the previous 2D data. The file format
of three-dimensional images was never mentioned in the courses. After some research I found that there
was no standard for 3D images. The closest I came to a 3D image file was the raw format, but as web
resources emphasized, raw is not a file format [itkw]. Look at the raw file as a stream of data which only
can be interpreted with knowledge of how the slices are organized in the string.
Fortunately the itk filewriter filter supports metaheader (mhd) files. By specifying the data model in a
metaheader and writing the output to the metaheader the 3D data are organized as specified by the metafile.
Typically that means if the input is 28 slices of 256 × 256, I want the output to be 28 files of 2D images.
The file naming convention is set up in the metafile and then taken hand of automatically by the filewriter
filter. Once I got used to the metafile concept it proved to be very useful. Since each slice was in a easily
viewable file format I could use any image viewer to examine the results.
An efficient way to present the results from the 3D segmentation is to use a 3D-model. Implementing a
3D-viewer with the VTK-library was out of the scope of this thesis, however, a viewer was implemented
in Matlab. Matlab did not handle the metafile concept so there I had to go the way through a raw data
file. It does not seem to challenging at first glance but it actually took some thinking. While you are in the
meta file you can always be sure which slice you are in and what the correct x, y coordinates are. When
the datastream was reorganized in Matlab it was easy to verify if all voxels ended up next to their initial
neighbor. There was harder to verify that I had not mixed x− and y−direction and that they were not
mirrored. The filewriter filter stores raw 3D data organized as the text in a book, where first page represents
first slice and line two comes after line one. In comparison, Matlab reads data column by column. I created
a map of the data structures to solve the problem. This practice ensured that the first 3D-tumor presented
was not upside down or mirrored.
Working with 3D MR images was not straight forward. The parameters that worked well on 2D images did
not work at all for 3D images. In the two-dimensional domain it was quick to test out a number of different
parameter settings. When I moved up to three dimensions this changed dramatically. A 256 × 256 × 28
MR-slab of a patients liver took several minutes to segment.11 Adjusting 4 to 5 parameters to get a valid
segmentation turned out to be an optimization problem. After trying a large value of parameter sets, I
found out I had to try another approach. The threshold level set segmentation’s two parameters would be
the easiest ones to estimate. Except from the initialization of the surface an upper and lower threshold are
11 The computer used was a PC compatible with a 2.53GHz Pentium 4, 512 MB RAM, Red Hat Enterprice Linux WS, kernel 2.4.2127.0.4.EL.
27
5.3 Level set segmentation in 3D
5 IMPLEMENTATIONS AND RESULTS
Figure 19: The synthetic liver segmented in 3D with Threshold level set filter. Parameters: seedpoint =
(128, 150, 35), radius = 3.0, thresholds: L = 50, U = 100.
needed. If we knew the true mean values of liver and tumor those could be used to set the thresholds. Then
I came up with the idea to make a model of a liver, a synthetic MR image.
The synthetic MR image of the liver is built of a set of ellipses where semi-axes are cosine functions to
get a organ like form. A spherical tumor with darker gray values is placed inside the liver quite close to
the edge. Then diffusion is applied and finally Gaussian noise is added. The result looks quite like a MR
image. Figure 18 on the preceding page shows a slice of the synthetic MR volum. Now I could test out the
segmentation algorithms knowing the true mean values of tumor and liver. I also knew the exact center of
the tumor. Provided with these facts I managed to get a segmentation in 3D. In figure 20 on the next page
the segmented synthetic liver tumor is presented as a 3D surface. The spherical shape is preserved despite
the applied noise and diffusion.
With knowledge of which parameters suited the gray level values in the synthetic image I went back to the
real MR image. The new knowledge very soon lead to valid segmentation results. The initialization was
not critical to localization. With too large initial size the algorithm created some artifacts. The artifacts
disappeared as soon as the initial size was reduced to proper value. In figure 21 on page 30 we see the
segmented tumor from a real MR image. We can not know for sure if the organ like surface is correct or a
result of the algorithm. By applying more smoothing of the MR image before segmentation we will get a
smoother object. However, smoothing will be on the cost of details.
28
5 IMPLEMENTATIONS AND RESULTS
5.3 Level set segmentation in 3D
measure
DSC of volume
DSC in plane x=150
DSC in plane y=128
DSC in plane z=35
value
0.7854
0.8769
0.8758
0.8767
volume of ground truth
volume of segmented
volume r=12
volume r=10
7140
4624
7238
4189
Table 2: Evaluation of the segmentation of the synthetic tumor.
Figure 20: Segmentation result of the synthetic tumor with spherical shape. Threshold level set method,
parameters: seedpoint = (128, 150, 35), radius = 10, thresholds: L = 50, U = 70.
The synthetic tumor was compared with the ground truth using DSC. Ground truth was produced by not
using blurring and noise when creating the reference tumor. Further, the number of voxels were compared
and DSCs were also calculated for all planes in the cartesian system. Results are presented in table 2.
29
5.3 Level set segmentation in 3D
5 IMPLEMENTATIONS AND RESULTS
Figure 21: Tumor segmented from a real 3D MR image. Exam 2212 with Threshold level set method,
parameters: seedpoint = (98, 137, 14), radius = 12, thresholds: L = 50, U = 70.
Another tumor in the same image is larger, however difficult to segment as it is located against the liver
edge. Figure 22 on the facing page shows the result of two segmentations. The first execution fails and
segments three objects which are connected. The level set surface converged after 418 iterations and the
process took 87 seconds. The second segmentation attempt is successful when parameters are tuned to this
specific image. Segmentation finished after 167 iterations and 75 seconds.
30
5 IMPLEMENTATIONS AND RESULTS
5.3 Level set segmentation in 3D
(a)
(b)
Figure 22: Segmentation of tumor failed as the interface break through at two areas and forms two false
objects (a). With adjusted parameters the boundary stopped at the tumor edge (b). Exam 2212 with
Threshold level set method, parameters: seedpoint = (140, 167, 9), radius = 5, thresholds: L = 0,
respectively U = 58 for (a) and U = 55 for (b).
31
5.4 Watershed segmentation
5 IMPLEMENTATIONS AND RESULTS
(a)
(b)
(c)
(d)
(e)
(f)
Figure 23: Watershed segmentation of liver tumor. Input MR image (a), smoothed image (b), gradient
magnitude (c), watershed segmented with parameters as specified in table 3. Image oversegmented (d), no
tumor detected (e) and successful segmentation of tumor (f).
experiment
1st segmentation (d)
2nd segmentation (e)
3rd segmentation (f)
iterations
5
5
5
conductance
3.0
3.0
3.0
lower threshold
0.001
0.02
0.01
max. depth
0.20
0.20
0.20
Table 3: Parameters used in watershed segmentation.
5.4 Watershed segmentation
The ITK-framework provides a filter for watershed segmentation. The documentation recommends to use
an anisotropic diffusion filter. This filter will smooth out noise and finer structures but preserve major
edges. Since the developers of the filter have tested extensively on MR images I had no reason to think I
would come up with a better preprocessing.
The degree of smoothing is controlled by the number of iterations the filter runs. A conductance parameter
controls the process sensitivity to edge contrast. There is also a time step parameter that is recommended
to set to a maximum of 0.25 for 2D and 0.125 for 3D images. The output from the preprocessing was
pipelined into the watershed filter. There the process is controlled by two parameters, level and threshold.
• Level – controls watershed depth.
• Threshold – controls lower threshold of input.
Both parameters are given in percent [0.0–1.0] of max depth in input image. Figure 23 shows output from
preprocessing and three different executions. The darker spot in the center of image (a) is the tumor. Image
(d) and (e) show an over-segmented versus under-segmented image. A detected tumor is shown in image
(f). When suitable smoothing parameters were found the watershed filtering was not sensitive to parameter
32
5 IMPLEMENTATIONS AND RESULTS
method
snake
threshold level set
watershed
5.5 Evaluation of results
parameters
r= 4.0
r= 6.0
r=10.0
r=14.5
u=135
u=140
u=145
t=0.01, d=0.20
t=0.001, d=0.20
t=0.02, d=0.18
DSC
0.756
0.916
0.944
0.903
0.902
0.916
0.899
0.896
0.896
0.896
time (s)
1.24
1.32
1.17
1.18
1.36
2.23
1.31
10.20
11.20
9.80
Table 4: Comparing accuracy of three segmentation methods with different parameter settings. Parameter
abbreviations: r=radius, u=upper threshold, t=lower threshold and d=max. depth.
setting. I got similar results with lower thresholds in the range 0.001–0.02 and maximum depth in the range
0.18–0.23. I found that a 0.125 time step was useful. The option to use Principal components were turned
off.
The three watershed outputs were generated with the parameters given in table 3 on the facing page. No
initializations were needed at all to produce these segmentations.
5.5 Evaluation of results
The accuracy of the segmentations were evaluated with Dice similarity coefficient. Given the limitations
of not providing a golden truth segmentation I will nevertheless compare my segmentation results against
an estimated truth. For real MR images we can estimate a truth of tissue classes by segmenting the tumor
manually. To establish a ground truth for the tissue classes I received help from a radiologist to perform
manual segmentation. As a result of the slice by slice segmentation of the watershed and snake algorithms,
the segmentation results were evaluated on slices rather than volumes. The results are presented in table 4.
• The snake was set with the following parameters:
-
GVF regularization coefficient = 0.20
number of iterations computed = 80
center of snake initializing (135, 135)
elasticity parameter α = 0.05
rigidity parameter β = 0
viscosity parameter γ = 1
external force weight κ = 0.5
iteration of snake-deforms iter = 20
• Threshold level set parameters
- center of interface initialization (seedX, seedY ) = (135, 135)
- initial distance r = 5.0
- lower threshold L = 100
• Watershed parameters
- conductance = 3.0
- diffusion iterations = 5
- principal components turned off
33
5.5 Evaluation of results
5 IMPLEMENTATIONS AND RESULTS
34
6 DISCUSSION
6 Discussion
In duration of the work with this thesis I have frequently been asked whether or not it is possible to segment
liver tumors in MR-images. For those of us working within the field of image processing, a few images will
give us an idea of the possibilities available. The tumors are pronounced quite clearly from the background
and the edges are not distinct but rather blurred. The significant difference in intensity of the object of
interest and background increases the chances for a successful segmentation. How challenging will it be
to make an automatic or semi-automatic segmentation algorithm? Unfortunately, this question can not be
answered in general terms as some aspects of this segmentation task is quite straight forward while others
are more challenging and requires a determent attitude to locate and identify the solutions.
6.1
Image features, the foundation for segmentation
Segmentation is based upon one or more features from the image itself. The most typical features are
color, intensity and texture, but can also include a variety of other features. Some applications make use
of features that is not directly accessible in the image itself, but need to be calculated from the directly
available information in the image. For example statistical properties or combinations of colors can also be
used. In image segmentation it is very important to consider all features in an image before you choose a
segmentation method. Selecting the right combination of features can make greater effect than just making
the algorithm as clever as possible. In the case of MR images of the liver I found that there is a one-band
or intensity image.12 In addition to variable intensity an intensity image may contain areas with textures.
Textures are excellent features for automatic segmentation when the area to segment has a texture that
differs significantly from the background and the area to segment is large enough compared to the pattern
in the texture. However in the case of MR images of the liver this is not applicable. The liver has a specific
texture that differs from the tumors texture. But the tumor is small compared to the “pattern repetition
rate” in the texture of the liver. If we managed to detect the tumor based on changes in texture itself, the
edges of the tumor would be drawn casually. Statistical properties will also require a background and a
foreground area which is large enough to provide valid results. If we used all voxels in the liver we would
get a sufficient number for calculating statistical properties of the liver tissue. However the number of
voxels in a typical 3 to 5 cm tumor is below an acceptable level for doing statistics.
Furthermore, it is possible to store MR-images with phase information. Phase information can be used to
provide thermal information in cryo-therapy and RF-ablation. The author of this thesis thought it could be
interesting to identify whether or not there is any correlation between phase and the segmented tumor.
Using phase was discussed with experienced MR-personal and we concluded it was not worth investigating
this as part of this report. First there would be very little phase difference from liver and tumor. A second
moment was that available images did not have phase information. My theories of using phase information
was inspired by newly acquired knowledge in principal component analysis (PCA) from a pattern recognition course. As my understanding regards to this academic field developed, I realized that the complex
systems of MR machines would already use PCA if it contributed to distinguish different tissues.
The eliminations above leave me with two useful features, intensity and topology. Intensity is the most
obvious feature of these and probably the most used property in image processing. Using intensity by itself
did not produce valid segmentation results, however when intensity was combined with topology we could
identify some useful results.
6.2 Promissing aspects for successful segmentation
I feel it will be appropriate to begin by focusing on those aspects of liver tumors that make the automatic
segmentation promissing. First, segmentation has been done manually on similar images. Further, you can
set up a set of rules for how to do the segmentation manually and the information is already in a digital
format. Based on these few statements alone a software engineer would say it is a typical task for computer
12 Standard
color images are a composite of three color bands where each band can be interpreted as an intensity image.
35
6.3 Aspects that complicates segmentation
6 DISCUSSION
automation. To view this in more depth it would be necessary to develop and increase our understanding
of image properties in detail.
We know the tumor appear as a darker area than the liver in general. Further, I have learned that the shape of
the tumor is a “lump-like object”, and other shapes such as ’C’, “a star” or “a worm” are not likely to appear.
The theory of image segmentation presents a number of methods to perform segmentation under these
conditions [Son, Eff00]. Often a handful of methods are able to produce a successful segmentation. When
the image is degraded by noise, artifacts, blurred by motion or other, this complicates the process. With
careful selection of preprocessing, one of the methods can hopefully handle the segmentation. There will
be situations where it is almost impossible to get a proper result no matter how we combine preprocessing
methods and segmentation methods.
The probability of being able to do a valid segmentation depends on the information content in the image.
An image degraded heavily by noise will be judged to provide little information. The same apply for other
distortions like unsharp images due to movement during image capture. Here the algorithms came us to
rescue. Even if an image appear very noisy, we might be able to extract a lot of information. A trained
human eye can also be able to withdraw information out of the noise. Both ultrasound and x-ray images
are good examples of the latter where the doctors find issues the untrained eye will not.
The results indicates the liver MR images have a high degree of information. As metioned in an earlier
section I was allowed to sort out images that was distorted by motion. Those were considered “bad images”
and in my case we would capture a new image. The noise level in the MR image is very low. The variations
inside an organ is mainly caused by natural variations in the tissue and dominates the noise from the MR
scanner completely. There can be artifacts from major vessels but they also provide information. Summarizing this I would claim that a MR image is characterized by providing a large amount of information.
This lead us to next section where I discuss the consequences of all the information available.
6.3
Aspects that complicates segmentation
All the promissing aspects found in the previous section might lead the reader to think that the segmentation
would be a relatively straight forward task. However, a large amount of information does not guarantee
good segmentation results.
The challenge in segmenting liver tumors is an exercise in handling a lot of information rather than handling a lack of information. The edge of the tumor is quite marked out thanks to the contrast agent. The
problem is that the tumor is surrounded by tissue with speckle, veins and vessels. The human eye can
see which edges belongs to which anatomical parts. This ability is hard to transfer to an algorithm. The
experimentation I have performed show that the problems are possible to reduce enough to be handled by
an algorithm. More on this in the following section.
6.4
Required preprocessing
First of all comes the study of required preprocessing. Is preprocessing needed at all and if, what sort of and
how much should we apply? Most algorithms require some sort of smoothing or has a built in smoothing
ability. There are different ways of smoothing and my studies indicates that the best segmentation is a result
of careful selection of smoothing and choise of algorithm. Other preprocessing to consider is adjusting for
variable background lighting. Filters for reducing the effect of blurring caused by movement are described
previously [Ros82]. As stated earlier, I was allowed to assume an image of satisfying quality. The liver is
segmented in another software component. This saves me for preprocessing to reduce problems near the
liver edges.
None of the algorithms I tested worked well without some smoothing. For others it might seem illogical to
obtain high resolution medical images and then the next step is to throw away all the details in a smoothing
filter. This may be partly true and I admit that it took me further studying to understand this myself. In
many applications a smoothing filter is a very efficient way to sort out the most crucial information in an
image. Because you have a smoothed copy of your image does not mean that the original is spoiled in case
36
6 DISCUSSION
6.5 The methods evaluated
your algorithm will check against the original image. Rather than throwing away information, smoothing
can be seen as spreading the information over multiple pixels in a way that suits the following algorithm.
When it comes to adjustment for variable background lighting this was only optional. Most images I
worked with had no gain in applying correction for variable background lighting. This is not a surprising
discovery. The algorithms all made use of the gradient one way or another. The gradient is little affected
by the slowly changing background variations. In some images dark areas appeared near the edges of the
image. The radiologist described those images as dark and bad images. I did not implement background
adjusting in my ITK-filter chain but I did some testing in Matlab which verified that some images could be
saved with certain degree of background adjustment.
6.5
The methods evaluated
The results show that the GVF snake scores around 0.9 on Dice similarity coefficient (DSC) which indicates
successful segmentation. Both inside and outside initializing lead to consistently high DSCs. When the
initialization circle is reduced to r = 4.0 the snake converges to a false edge inside the tumor giving a
low DSC score. A DSCs above 0.7 is considered acceptable [Zou04], however the results show that best
results are achieved with initialization close to the inside of the tumor edge. The snake was quickest of the
implemented algorithms, even though the handicap of being implemented in Matlab. Xu suggested to run
100 iterations, however, my results show convergence with 20 iterations and this explains the fast execution
times. The parameters to model the snake was not critical. Experiments where parameters were scaled by
1/3 and 3 gave identical results.
The threshold level set algorithm attained DSCs similar to the snake. Unlike the snake, good results were
measured also when initialized with a small circle in the center of the tumor. Nevertheless the choise
of the upper threshold will have direct implications to the results. Anyway, the results show successful
segmentation with upper level in the range 135 − 145. The execution times are up to twice of what the
snake performs.
The strength with the watershed method is very consistent results over a range of parameter settings. As
the results indicate, the lower threshold works in the range 0.001 − 0.02. By using a value near the middle,
the range of maximum depth will also increase. Another strength is when oversegmentation of the liver
occurs, the tumor is still segmented with the same precision. The drawback with the need to locate the
region of interest after the segmentation is outweighed by the fact that no initialization is required. The
experiments show the watershed algorithm lack of speed with execution times around 10 seconds. This is
a result of the large images used.
6.6 Robustness of algorithms
In engineering there is seldom one method collects the best from different directions without also having
some drawbacks and the same yields for image processing. Those algorithms that managed to create
the most convincing results were also those that had the tightest requirements in parameter settings. On
the other hand, robustness is more than tolerance for parameter variations. The main issues in algorithm
robustness are:
• Tolerances to parameter settings.
• Sensitivity to segmentation near other edges.
• Requirements to accurate initialization.
• Sensitivity to image noise.
• Sensitivity to artifacts and image distortions.
• Handling of weak edges.
37
6.7 Prefered method
6 DISCUSSION
The signal to noise ratio is very high in MR images. Generally we will be concerned by how algorithms
react to noise in the input image. The noise level is very low and we always apply some smoothing which
will reduce the noise level additionally. There is some quantization noise as the input image uses 16-bits
integer values to represent intensity. When reading the image file, the first step is to convert the data to
floating point for the highest possible precision in the filtering process. The use of floating point precision
allows us to neglect the quantization noise. Whether the used algorithms are sensitive to noise has little
relevance to the application of this thesis.
GVF snakes were very tolerant to parameter settings for elasticity and rigidity of the snake. The snake
failed if it was initialized to close to other edges than the tumor. When initialized inside the tumor the
snake did not require an accurate initialization, but had to be initialized around the densest part of the
tumor. When initialized outside the snake was very sensitive to all sort of edges. Weak edges are currently
handled very well by the traditional snake and well by the GVF snake. For GVF snakes there become a
trade off of smoothing to ensure convergence to edge and correctness of edge localization.
The threshold level set method requires appropriate parameter settings for upper and lower threshold. These
parameters were easy to calculate from the mean value of liver intensity and tumor intensity. The method
was medium sensitive to accuracy in initialization. Artifacts did not disturb the result as long as I initialized
inside the tumor. Furthermore, other edges were a potential problem, however week edges were handled
very well.
The watershed segmentation method was very robust and when useful parameters were found they seemed
to work well in a variety of liver images. Automatic operation is a plus and no initialization was required
at all. Other edges that did not form a catchment basin did not cause problems. Artifacts and veins are
segmented equivalent to the tumor without disturbing the segmentation of the tumor. Unfortunately weak
edges are not handled well.
Image segmentation can be seen as a two class problem in pattern recognition. The segmentation algorithm
will then be the equivalent to the classifier. A classifier that is tuned for optimal results on the training set
will often fail on new samples. We say the classifier is over trained, meaning it has become too specialized
to the laboratory data and less useful for real world data [Duda01]. Over training is a trap it is easy to fall
into and over training can lead to horrible results when the algorithm is plugged into a real world system.
6.7 Prefered method
The software components I have written in my thesis are pieces of programs with enough functionality to
test and verify the selected methods. Next step will be to collect these programs in a working module in
the IGVAC13 system. Then I have to chose one of the three methods.
Snakes worked fine when well initialized. It was difficult to find a good way to connect information
between the image slices, and I did not come up with any well working algorithm. The way the level set
methods handle three dimensions was very elegant compared to what I could come up with in the case
of the snake. The threshold level set method has strict requirements to the threshold parameters, but both
upper and lower threshold are easy to calculate based on the initialization of the tumor location. Watershed
segmentation had the advantage of segmenting the liver interior without any initialization. There were
two drawbacks with the watershed algorithm. First, you had to point out the object of interest after the
segmentation. Second, there is a risk that the watershed algorithm does not find anything if the tumor
edges are too weak. I would not implement watershed segmentation without a second segmentation system
as a backup.
When I weigh segmentation results, robustness and ease of use, based on testing and results from the work
with this thesis, I recommend to use the threshold level set method. Unfortunately I did not manage to run
the other level set methods on 3D images. The ITK-library is under active development so the run-time
problem with the library might be solved in a later version. A contribution to the library development could
be in the form of a minimum program that are able to reproduce the run-time errors. I would probably have
13 Image Guidance, Visualization and Control – a software system used by the Interventional Centre at Rikshospitalet University
Hospital.
38
6 DISCUSSION
6.8 Preparing the segmentation output
to reduce the size of the input data as well to ease debugging for the library developer. Writing the needed
debugging software can be a time consuming task and I decided that was out of the scope of my thesis.
When developing a library, a great part of the total programming effort is to write debugging and test code
[Str93].
6.8 Preparing the segmentation output
The output from the level set segmentation is a binary image. Instead of just handle the binary image to
the next module I decided to integrate it with the input image. This will also reduce the risk that the binary
image get connected with wrong MR image in another module. The module I send my result to will map
a color and opaque coefficient to each intensity level in the MR image. By setting all tumor voxels to
maximum intensity they will have a unique gray level and can efficiently be marked with a suitable color.
The segmentation result is integrated with the MR image by
Iout = IMR − (IMR · Itumor ) + k · Itumor
(19)
where I is representing the images and k is maximum intensity. In the segmented image Itumor all tumor
voxels are ones and all other voxels are zeros.
39
6.8 Preparing the segmentation output
6 DISCUSSION
40
7 CONCLUSION
7 Conclusion
7.1
Summary
Automatic or semi-automatic segmentation tools are essential software tools for efficient use of MR images
in surgery. Different segmentation methods have been used with success in tools based on the Insight
Toolkit library. Previous work has mainly focused on segmentation of organs or segmenting organs into
their subparts.
Little was written about segmentation of liver tumors from MRI images by the time I started the work on
this thesis. This thesis proves that methods aimed at general purpose segmentation tools in medical imaging
can be used for automatic segmentation of liver tumors. The quality of the segmentations were similar to
manual segmentation and will speed up segmentation in operative imaging.
Among the segmentation methods investigated in this thesis the level set method segmented three-dimensional
MR images with best results given it was initialized from the inside. GVF-snakes, snakes steered by a Gradient Vector Flow field, was also used with success when initialized sufficiently close to the tumor edge.
Watershed segmentation, a classic in image segmentation, marked out as the most automatic method of the
three.
I found that the snake and the level set method were best initialized from the inside of the tumor. Then
successful segmentation was obtained with less smoothing and better preservation of details.
7.2 Future work
In addition to collecting my programs to a plug-in module in the hospitals system (IGVAC) the program
need to be extend to handle segmentation with a probe through the tumor. This can be done by segmenting
the probe or in combination with tracking information which provides the position of the probe [Grim].
Improved robustness can be gained by segmenting the blood vessels inside the liver during preoperative
image analysis. With the use of image registration the vessels can be found in the operative images and
eliminate them from the feature map used in tumor segmentation. Such an extension also contributes to
added complexity and there is no guarantee the added feature will increase the robustness of the complete
system.
The user interface in the main application must be extended to allow activation of the segmentation and to
collect initialization points from a pointing device and transfer them to the segmentation module. Finally
the main program must receive the segmented image and present the image as an opaque volume.
The next step will then be to connect the information about the tumor with the monitoring of the cryoor RF-ablation. The program should verify that all tumor cells are inside an ablation zone providing a
satisfactory ablative margin.
41
7.2 Future work
7 CONCLUSION
Acknowledgements
I would like to thank my supervisor Eigil Samset for arranging the project, for guidance during the process and for valuable feedback on the thesis. Further I appreciate the help provided by Per Kristian Hol,
interpreting the MR images, and Ole Jakob Elle for his daily encouragements and interest for the students
work.
Finally, I would thank my family for the patience and understanding over the last two years, and not to
forget the dog which pulls me away from the computer, up to the local ridges we both enjoy so much.
42
REFERENCES
REFERENCES
References
[Sam01] Eigil Samset et al. Validation of estimated 3D temperature maps during hepatic cryo surgery.,
Elsevier, Magnetic Resonance Imaging 19 (2001), pp715-721.
[ummw] Holly Strawbridge, University of Miami Medicine Online, http://www.miami.edu/ummedicinemagazine/summer2001/radiology.html
[Heu03] Håkon Heuch, Segmentation of the liver from MR and CT images Cand.sci. thesis, NTNU, June
12th, 2003.
[Xu97] C. Xu and J.L. Prince Gradient Vector Flow: A New External Force for Snakes IEEE Proc. Conf
on Comp. Vis. Patt. Recog. (CVPR’97) p66-71.
[Xu00] C. Xu, A. Yezzi and J.L. Prince, On the Relationship between Parametric and Geometic Active
Contours In Proc. of 34th. Asilomar Conference on Signals, Systems and Computers p483-489,
Oct 2000.
[imaw] Imaginis, History of Medical Diagnosis and Diagnostic Imaging http://www.imaginis.com
[Hae01] Dieter Haemmerich, Hepatic Bipolar Radio-Frequency Ablation Between Separated Multiprong
Electrodes,IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 48, NO. 10, OCTOBER 2001, pp1145-1152.
[ritw] Ph.D. Joseph P. Hornak, The Basics of NMR http://www.cis.rit.edu/htbooks/nmr/
[ucsdw] University of California San Diego,
http://health.ucsd.edu/mri/pi_apps.asp
Applications and Clinical Benefits of MRI
[Bell] Robert A. Bell Ph.D. Magnetic Resonance in Medicine in 2020 Image Economics, December 2004,
http://www.imageeconomics.com
[medw] MedicineNet, Radiofrequency ablation, http://www.medterms.com/
[Kas] M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models, Int. J. Computer Vision,
1(4):321-331, 1987.
[Son] M. Sonka, V. Hlavac and R. Boyle: Image Processing, Analysis, and Machine Vision, 1999.
Brooks/Cole Publishing Company.
[wolf05] Mathworld, Differential Geometry of Curves http://mathworld.wolfram.com/Curvature.html
[Gray97] A. Gray, Modern Differential Geometry of Curves and Surfaces with Mathematica, 2nd ed.,
1997, Boca Raton, FL: CRC Press.
[Osh88] S. Osher, J.A. Sethian, Fronts Propagating with Curvature Dependent Speed: Algorithms Based
on Hamilton-Jacobi Formulation, Journal of Computational Physics, Vol 79, pp.12-49, 1988.
[Ada95] D. Adalsteinsson, J.A. Sethian, A Fast Level Set Method for Propagating Interfaces, Journal of
Computational Physics, Vol 118(2), pp.269-277, May, 1995.
[Set95] J.A. Sethian A Fast Marching Level Set Method for Monotonically Advancing Fronts. 1995.
[Vin91] L. Vincent and P. Soille, Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersioin Simulations, IEEE, Transaction on pattern analysis and machine intelligence. vol 13, no 6,
June 1991, p 583-598.
[xuw] Chenyang Xu Active Contours,
http://iacl.ece.jhu.edu/projects/gvf
Deformable
43
Models,
and
Gradient
Vector
Flow
REFERENCES
REFERENCES
[itkw] Insight Toolkit, National Library of Medicine Insight Segmentation and Registration Toolkit
http://www.itk.org
[vtkw] Visualization Toolkit, An open-source system for 3D graphics, image processing, visualization,
computational geometry, and volume rendering. http://www.vtk.org
[Zou04] K.H. Zou, S.K. Warfield, A. Bharatha, et al. Statistical validation of image segmentation quality
based on spatial overlap index, Acad Radiol 2004; 11:pp178-189.
[Eff00] Nick Efford, Digital Image Processing, a practical introduction using Java, 2000 Person, Addison
Wesley, ISBN 0-201-59623-7.
[Ros82] A. Rosenfeld and A. C. Kak, Digital Picture Processing, Academic Press, New York, 2nd edition,
1982.
[Duda01] R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, Second ed. 2001, Wiley.
[Str93] Bjarne Stroustrup, The C++ Programming Language, Second ed. 1993, Addison Wesley, ISBN
0-201-53992-6.
[Grim] E.
Grimson
et
al.Clinical
Experience
with
a
High
Precision
Imageguided Neurosurgery System?,
MIT AI Laboratory,
Cambridge MA, USA,
http://www.ai.mit.edu/people/leventon/Research/9810-MICCAI-Surgery/
44