Segmentation In The Field
Medicine
Advanced Image Processing course
By: Ibrahim Jubran
Presented To: Prof. Hagit Hel-Or
What we will go through today
• A little inspiration.
• Medical image segmentation methods:
- Deformable Models.
- Markov Random Fields.
• Results.
Why Let A Human Do It, When The
Computer Does It Better?
• “Image data is of immense practical importance
in medical informatics.”
• For instance: CAT, MRI, CT, X-Ray, Ultrasound.
All represented as images, and as images, they
can be processed to extract meaningful
information such as: volume, shape, motion of
organs, layers, or to detect any abnormalities.
Why Let A Human Do It, When The
Computer Does It Better? Cont.
• Here’s a task for you:
Look at this image:
could you manually mark the
boundaries of the two
abnormal regions?
Answer: Maybe…
Not Bad...
And… What if I told you to do it in 3D?
Answer?
You would probably fail badly.
But… the computer, on other hand, dealt
with it perfectly:
Common Methods:
Deformable Models
• Deformable models are curves whose deformations
•
•
are determined by the displacement of a discrete
number of control points along the curve.
Advantage: usually very fast convergence,
depending on the predetermined number of control
points.
Disadvantage: Topology dependent: a model can
capture only one ROI, therefore in images with
multiple ROIs we need to initialize multiple models.
Deformable models
• A widely used method in the medicine field is the
Deformable Models, which is divided into two
main categories:
- The Parametric Deformable Models.
- The Geometric Deformable Models.
• We shall discuss each of them briefly.
Geometric Models
• Geometric Models use a distance transformation
to define the shape from the n-dimentional to an
n+1-dimentional domain (where n=1 for curves,
n=2 for surfaces on the image plane…)
Example of a transformation
• Here you see a transformation from 1D to 2D.
Geometric Models cont.
• Advantages:
1) The evolving interface can be described by
a single function even if it consists of more
than one curve.
2) The shape can be defined in a domain with
dimensionality similar to the dataset space (for
example, for 2D segmentation, a curve is
transformed into a 2D surface) -> more
mathematically straightforward integration of
shape and appearance.
In Other Words…
• We transform the n dimensional image into an
n+1 dimensional image, then we try to find the
best position for a “plane” , called the “zero level
set”, to be in.
• We start from the highest point and descend,
until the change in the gradient is below a
predefined threshold.
And Formally…
• The distance function:
1
𝐸 𝐶 =
0
𝑔(|𝛻𝐼(𝐶 𝑠 )|) 𝐶 ′ 𝑠 𝑑𝑠
• g is the speed function, C is our zero level set
• C’ forces the boundaries to be smooth.
Geometric Deformable Models
Example
Geometric Models Results
Geometric Deformable Models
Short demonstration
Click to watch a demonstration of the MRF
Parametric Models
• Also known as “Active contours”, or Snakes.
Sounds familiar?
• The following slides are taken from Saar Arbel’s
presentation about Snakes.
Five instances of the evolution of a region based deformable model
A framework for drawing an object outline
from a possibly noisy 2D image.
An energy-minimizing curve guided by external
constraint forces and influenced by image forces
that pull it towards features (lines, edges).
Represents an object boundary or some other
salient image feature as a parametric curve
External Energy Function
Internal Energy Function
A set of k points (in the discreet world)
or a continuous function that will
represent the points
Snakes are autonomous and self-adapting in
their search for a minimal energy state
They can be easily manipulated using external
image forces
They can be used to track dynamic objects in
temporal as well as the spatial dimensions
Common Methods:
Learned Based Classification
• Learning based pixel and region classification is
among the popular approaches for image
segmentation.
• Those methods use the advantages of supervised
learning (training from examples) to assign a
probability for each image site of belonging to
the region of interest (ROI).
The MRF & The Cartoon Model
A cartoon model
The Markov Random Field
• The name “Markov Random Field” might sound like a
hard and scary subject at first… I thought so too when I
started reading about it…
•
Unfortunately I still do.
An unrelated photo of Homer
Simpson
• Click to watch a demonstration of the MRF
• https://www.youtube.com/watch?v=hfOfAqLWo5c
The MRF & The Cartoon Model
• The MRF uses a model called the “cartoon
model”, which assumes that the “world” consists
of regions where low level features change
slowly, but across the boundaries these features
change abruptly.
• Our goal is: to find 𝜔, a “cartoon”, which is a
simplified version of the input image but with
Labels attached to the regions.
𝜔 & The Cartoon Model
𝜔𝑠 is modeled as a discrete random variable
taking values in 𝛬 = {1, 2, … , 𝐿}.
The Cartoon Model Cont.
• The discontinuities between those regions form a
curve 𝛤 (the contour).
• (𝜔, 𝛤) form a segmentation.
• We will only focus on finding the best 𝜔, because
once 𝜔 is determined, 𝛤 can be easily obtained.
More Cartoon Model Examples
Original
labelled
(𝜔, 𝛤)
The Probabilistic Approach
For Finding The Model
• For each possible segmentation / cartoon of the
input image G we want to give a probability
measure that describes how suitable the cartoon
is, for this specific image.
• Let 𝛺 be the set of all possible segmentations,
Note that 𝛺 is finite!
The Probabilistic Approach cont.
• Assumptions: in this approach we assume that
we have 2 sets of variables:
1) The observation random variables Y.
ℱ ∈Y ,the observation , represents the lowlevel features in the image.
2) The hidden random variables X.
The hidden entity 𝜔 ∈ X represents the
segmentation itself.
Observation and Hidden Variables
𝜔
Low level features,
for example:
𝑔𝑟𝑎𝑦 𝑠𝑐𝑎𝑙𝑒
𝑡𝑒𝑥𝑡𝑢𝑟𝑒
𝑏𝑟𝑖𝑔ℎ𝑡𝑛𝑒𝑠𝑠
Defining the Parameters needed
• First we need to define how well a segmentation
𝜔 fits the image features ℱ.
P(𝜔 | ℱ) – the image model.
• We want every image to posses a set of
properties.
P(𝜔) – the prior, tells us how well 𝜔 satisfies
these properties.
Illustration Of P(𝜔 | ℱ)
original
P(𝜔 | ℱ) is high
P(𝜔 | ℱ) is low
Example
• We want the regions to be more homogeneous.
For example,
In this image P(𝝎)
would be a large
number
Example cont.
But, In this image
P(𝝎) would be a
very small number
Our Goal
• Our goal is to maximize P(𝜔|ℱ), since the higher
this probability is, the more suitable the
segmentation 𝜔 fits the image features ℱ.
An unrelated photo of Homer
Simpson (again)
• Click to watch a demonstration of the MRF
A Lesson In Probability
• As you might remember (probably not) from
Probability lectures,
P(ℱ|𝜔)∗P(𝜔)
P(𝜔|ℱ) =
𝑃 ℱ
• Since 𝑃
ℱ is constant for each image and so is
dropped, therefor, we are looking for 𝜔 that
maximizes the posterior.
Defining the Parameters needed Cont.
• In addition to the probability distributions that
we defined, our model also depend on certain
parameters that we denote by 𝛩.
• In the supervised segmentation we assume these
parameters are either known or that a training
set is available.
• In the unsupervised case, we will have to infer
both 𝜔 and 𝛩 from the observable entity ℱ.
The MRF cont.
• There are many features that one can take as
observation ℱ for the segmentation process:
gray-level, color, motion, different texture
features, etc.
• In our lesson we would be using a combination of
classical, gray-level based, texture features and
color, instead of direct modeling of color
textures.
Feature extraction
• For each pixel s, we define a vector 𝑓𝑠 , which
represents the features at that pixel.
• The set of all the feature vectors form a vector
field:
ℱ={𝑓𝑠 | s ∈ S}, S = {𝑠1 , 𝑠2 , 𝑠3 , , … 𝑠𝑁 } (pixels).
And as you remember, ℱ is the Observation, and
will be the input of the MRF segmentation
algorithm.
Notes
• REMINDER: our features will be texture and
color.
• We use the CIE-L*U*V color plane, so regions will
be formed where both features are
homogeneous while boundaries will be present
where there is discontinuity in either color or
texture.
CIE-L*u*v* VS. RGB
CIELUV color histogram
RGB color histogram
The Markov Random Field
Segmentation Model
• Lets start by defining P(𝜔):
1 −
P(𝜔) = 𝑒
𝑍
Let’s call this SQUIRREL
𝐶𝜖𝑙 𝑉𝐶 (𝐶 )
• 𝑙 = 𝑠𝑒𝑡 𝑜𝑓 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟𝑖𝑛𝑔 𝑝𝑖𝑥𝑒𝑙𝑠.
• We defined P(𝜔) in a way that it represents the simple
fact that segmentation should be locally homogeneous.
Definitions
•𝑍 =
𝑤𝜖𝛺
1 = The number of possible
cartoons.
• 𝑣𝑐 = 𝛿
𝜔𝑠 , 𝜔𝑟
+1 𝑖𝑓 𝜔𝑠 ≠ 𝜔𝑟
=
−1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
• 𝜔𝑠 = The label of pixel s
And now… the FUN part !!
•
Don’t listen to me, just RUN!
The Image Process
• We assume P(𝑓𝑠 |𝜔𝑠 ) follows a normal distribution
N(𝜇𝑠 , 𝛴).
• N(𝜇𝑠 , 𝛴) =
1
2𝜋
𝑛 |𝛴
𝜆|
exp
1
−
2
𝑓−
𝜇𝜆 𝛴𝜆−1
𝑓 − 𝜇𝜆
T
The Image Process cont.
• n = The dimension of our color-texture space.
• 𝜆 = A pixel class.
• 𝜇𝜆 = The mean vector (The average of all the feature
vectors within the class 𝜆).
• 𝛴𝜆
= The covariance matrix, which describes the
correlation between each two features in a given
class.
Intuition
𝑔𝑟𝑎𝑦 𝑠𝑐𝑎𝑙𝑒
𝑡𝑒𝑥𝑡𝑢𝑟𝑒
Our feature vectors would look like this:
𝑏𝑟𝑖𝑔ℎ𝑡𝑛𝑒𝑠𝑠
1
190
2
180
𝜇1
205
2
200
2
220
2
220
4
𝜇2
3
30
5
25
20
5
30
10
5
35
6
130
6
140
𝜇3
117
6
140
5
104
6
140
Intuition cont.
• For example: if we want to include vector 1 in class 𝜇2
1
then: −
2
𝑓
− 𝜇𝜆 𝛴𝜆−1
𝑓 − 𝜇𝜆
T
would be a very small
number, and so P(𝑓𝑠 |𝜔𝑠 ) is very low.
• If we include vector 5 in class 𝜇3 then − 12
𝑓 − 𝜇𝜆 𝛴𝜆−1 𝑓
The Image Process cont.
• We assume each pixel feature is independent, and so:
P(ℱ|𝜔) =
P(𝑓𝑠 |𝜔𝑠 )
𝑠∈𝑆
1
=
𝑠∈𝑆
2𝜋 𝑛 |𝛴𝜔𝑠 |
Let’s call this DOG
1
exp − 𝑓 − 𝜇𝜔𝑠 𝛴𝜆−1 𝑓 − 𝜇𝜔𝑠
2
Let’s call this CAT
T
Fun Equations cont.
•
P(ℱ|ω)∗P(ω)
REMINDER: P(𝜔|ℱ) =
(but we drop P(ℱ))
P(ℱ)
=
(𝐷𝑂𝐺 ∗ exp 𝐶𝐴𝑇 )
𝑠∈𝑆
𝐿𝑂𝐺: =
(ln
𝑠∈𝑆
1
1
∗
∗ exp
𝑍
+ 𝐶𝐴𝑇) − 𝛽 ∗
2𝜋 𝑛 |𝛴𝜔𝑠 |
=⋯=
𝑆𝑄𝑈𝐼𝑅𝑅𝐸𝐿
𝑆𝑄𝑈𝐼𝑅𝑅𝐸𝐿
Fun Equations cont.
=−
ln
𝑠∈𝑆
−𝛽 ∗
1
𝑛
2𝜋 |𝛴𝜔𝑠 | + 𝑓 − 𝜇𝜔𝑠 𝛴𝜆−1 𝑓 − 𝜇𝜔𝑠
2
T
𝛿 𝜔𝑠 , 𝜔𝑟
{𝑠,𝑟}∈𝑙
• REMINDER: We need to find 𝜔 that MAXIMIZES this expression.
• Or, equivalently, we need to find 𝜔 that MINIMIZES the
expression inside the “-”.
MINIMIZATION
• There are two main methods used to minimize
our expression:
1) ICM (Iterated Conditional Modes).
2) Gibbs sampler.
• In some of the results we would be comparing
those two methods.
Parameter estimation
• There are some parameters in our equations that
should be estimated, with or without supervision:
1) If a training set is provided, then those
parameters can be easily calculated based on the
given data.
2) If we do not have such a training set, we would
have to use an iterative EM algorithm.
Supervised Parameter Estimation
cont.
• We can estimate 𝜇𝜆 by summing the vectors of
class 𝜆 in the given dataset (and normalize the
result).
• We can estimate 𝛴𝜆
by summing 𝑋𝑖 −
Unsupervised Parameter
Estimation cont.
• When no dataset is available we would be using
the Estimation Maximization for Guassian
Mixture Identification.
• Our goal is to find parameter values which
maximize the normalized log-likelihood function
(which is a concave function).
1
𝐿=
𝐷
𝐷
log(
𝑖=1
𝑃(𝜆|𝑑𝑖 ))
𝜆∈𝛬
The EM Algorithm
• E step: compute a distribution on the labels based on the
current parameter estimates.
• M step: calculating the parameters again based on the
new labels, very similar to the supervised case.
• We repeat those two steps until convergence.
• K-Means is a specific case of the EM algorithm.
• The EM approach is similar to the Gradient Descent.
MRF Results (Supervised)
Texture
ICM
Gibbs Sampler
Color
Combined
MRF Results (Unsupervised)
Texture
ICM
Gibbs Sampler
Color
Combined
The Finale
• Segmentation in the medicine field covers many topics
and methods, today we covered 2 of them, saw some
results and introduced a small estimation algorithm
widely used in those topics.
References
•
•
•
•
•
•
A Markov random field image segmentation model for color
textured images. –Zoltan Kato, Ting-Chuen Pong.
Medical Image Segmentation . –Xiaolei Huang, Gavriil Tsechpenakis.
Deformable Model-Based Medical Image Segmentation. –Gavriil
Tsechpenakis.
http://en.wikipedia.org/wiki/Markov_random_field
Saar Arbel’s presentation about snakes.
http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_
algorithm