www.nr.no
INF 5300
Deformable contours, I
An introduction
Norsk Regnesentral,
9th of February 2005
Plan, lecture I
1.
2.
3.
What is segmentation?
Motivation, why deformable contours?
The basic formalism: snakes as presented by Kass and Witkin.
www.nr.no
Practical information
1.
Two double lectures about deformable contours.
2.
The lectures are based on articles, references will be given during the course.
3.
The transparencies will be available on the course website.
4.
The transparencies define the exam requirements www.nr.no
Plan, lecture II
1.
Matlab demonstration
2.
Variations of the basic snakes a. Topological adaptability b. 3D
3.
The capture range problem a. Distance measure based solutions b. Gradient vector flow field based solutions www.nr.no

What is segmentation?
1.
Sonka: “The main goal of image segmentation is to divide an image into parts that have a strong correlation with objects or areas of the world contained in the image”
2.
3.
Sonka: “Complete segmentation: results in a set of disjoint regions corresponding uniquely with objects in the input image.”
Sonka: “Partial segmentation: regions do not correspond directly with image objects.” www.nr.no
What is segmentation?
1.
Aurdal: “Segmentation is a process that maps an input image (possibly multispectral) into an output image where each pixel in the input image is given a label indicating its membership in a group of pixels sharing some common property. Group inclusion can be based on: a. Similar spectral properties.
b. Similar higher order spectral properties (texture).
c. etc…..
www.nr.no
What is segmentation?
1.
Sonka: “To achieve a complete segmentation, cooperation with higher processing levels which use specific knowledge of the problem domain is necessary.”
2.
Sonka: “There exists a whole class of segmentation problems that can be solved using lower level processing only.” www.nr.no
What is segmentation?
1.
2.
3.
4.
Segmentation is not the same thing as classification!
In segmentation each pixel receives a label indicating its membership in a group of pixels (group 1, group2 etc…)
In classification each pixel receives a label indicating its membership in a group of pixels having a sensible physical interpretation (water, ship, land etc…)
Classification will often use segmentation as a preprocessing step www.nr.no

Motivation
1.
2.
Classical assumption: Digital images will show real world objects as well defined regions with unique gray levels and a clear border against a uniform background.
a. This assumption can hold true in certain cases, but typically it does not.
Real world objects seen in digital images might very well blend with the background and many parts of the objects might be more or less occluded.
www.nr.no
Motivation
1.
2.
Classical assumption: The image contains all the information necessary in order to produce a good segmentation.
a. Again, this might be so in certain situations, but this is more of an exception than the rule.
In real world images we will typically need higher level information about the objects we are trying to segments: a. Size b. Shape c. Position www.nr.no
Motivation
Beware of extreme case of blending and occlusion www.nr.no
Motivation
1.
We know of many cases where such knowledge is used: a. Thesholding: We assume knowledge of graylevels….
b. Adaptive thesholding: We assume knowledge of the size of the objects we are interested in… c. Hough transform: We assume certain shapes… www.nr.no

Motivation
1.
The methods we will look at now are just another example of segmentation methods using models external to the image in order to obtain the best possible segmentation.
2.
Typical segmentation problems solved with this kind of image are for instance ultrasound cardiac images.
a. Much noise and blurred edges b. Much knowledge about shape etc.
www.nr.no
The initial idea: Snakes
1.
Snakes are model-based methods for localization and tracking of image structures.
2.
3.
4.
The snake is defined as an energy minimizing contour (often defined using splines).
The energy of the snake depends on its shape and location within the image.
Snakes are attracted to image boundaries through forces.
www.nr.no
Motivation
Left ventricle of the heart, apical view.© GE medical www.nr.no
The initial idea: Snakes
1.
2.
The approach is iterative: a. The user draws an initial approximate contour.
b. A dynamic simulation is started.
c. The contour is deformed until it reaches equilibrium.
Snakes depend on: a. Interaction with the user.
b. Interaction with a high level description.
c. Interaction with image data adjacent in space and time.
www.nr.no

The initial idea: Snakes
1.
2.
Different types of forces will affect the snake: a. Internal forces.
1) Tension forces that make the snake act like a membrane.
2) Rigidity forces that make the snake act like a thin plate that resists bending.
Image forces
3.
Constraint forces.
a. User supplied forces or forces that come from higher level image understanding processes.
www.nr.no
What is a parametric contour?
1.
2.
Let x(s)=cos(2
π s) and y(s)=sin(2
π s).
Furthermore, let s ∈ [0,1].
Then v (s) describes a circle as s varies from 0 to
1.
www.nr.no
The initial idea: Snakes
1.
The contour is represented as:
2.
This is a parametric representation of the contour.
3.
The vector describing the position of every point on the contour makes one pass over the entire contour as s varies from its minimum to its maximum value.
Typically s is normalized: www.nr.no
The initial idea: Snakes
1.
The shape of the contour v (s) is dictated by a functional:
2.
This functional can be seen as representing the energy along the contour.
3.
The final position of the contour will correspond to a minimum of this energy function.
www.nr.no

The initial idea: Snakes
1.
The first term of the functional can be viewed as a representation of the internal deformation energy:
The initial idea: Snakes
1.
The second term:
2.
imposes rigidity, changes in direction should be smooth.
www.nr.no
www.nr.no
The initial idea: Snakes
1.
The first term:
2.
imposes tension, that is, we want a curve as short as possible. www.nr.no
The initial idea: Snakes
1.
The weight functions w
1
(s) and w
2
(s) are used to modify the weights given to the two terms.
2.
Why do you think it is a smart idea to make these a function of s? Can you think of cases where this might be a nice feature.
www.nr.no

Physical analogy
1.
2.
3.
The first term makes v act like a membrane
The second term makes it act like a thin plate.
The weighting functions will just balance these two types of behavior.
The initial idea: Snakes
1.
The second term of the functional introduces an external energy term into the expression for the functional:
The initial idea: Snakes
1.
2.
In this expression, P( v (s)) denotes a scalar potential function defined on the image plane.
We obviously choose potential functions P(x,y) whose minima coincide with intensity extremas, edges or other interesting features of the image.
www.nr.no
www.nr.no
The initial idea: Snakes
1.
The most common way of defining P(x,y) is the following:
2.
What does this expression mean in terms of the final position of the snake.
www.nr.no
www.nr.no

The initial idea: Snakes
1.
A variation of this can be obtained by letting:
2.
What would be the use for this kind of definition?
The initial idea: Snakes
1.
Remember:
2.
Also:
3.
And: www.nr.no
www.nr.no
The initial idea: Snakes
1.
So now we know how we measure the goodness of the current snake position, we simply measure the
‘energy’ given by:
2.
We are basically looking for a minimum in this energy.
3.
When the minimum is reached, the snake will be in an optimal position.
4.
How do we find this minimum?
www.nr.no
The initial idea: Snakes
1.
Let assume that the functions w
1
(s) and w simply constants.
2
(s) are
2.
In practice, this is by far the most common situation.
3.
Thus we have: www.nr.no

The initial idea: Snakes
1.
The total energy can thus be written:
The initial idea: Snakes
1.
We have the equation:
2.
This becomes: www.nr.no
www.nr.no
The initial idea: Snakes
1.
It can be shown that the v that minimizes E must satisfy the following differential equation:
2.
This is the Euler-Lagrange equation for the given problem.
3.
You’ll find more on this in appendix A .
www.nr.no
The initial idea: Snakes
1.
We have:
2.
In component form this becomes: www.nr.no

The initial idea: Snakes
1.
2.
3.
In practical life the parameter domain is discrete, that is {s
0
=0, s
1
,s
2
,…,s n
=1}.
The curve v (s) will be described by n+1 sample points (control points), v
0
= v (s
0
),…, v i
= v (s i
),… v n
= v (s n
).
The differential equations in the previous slide must be transformed into discrete versions.
The initial idea: Snakes
1.
2.
3.
4.
We must solve these equations numerically.
That means discretizing them.
First we need to find expressions for the second and fourth derivatives of x and y with respect to s.
The details of this are discussed in appendix B .
www.nr.no
www.nr.no
The initial idea: Snakes www.nr.no
The initial idea: Snakes
1.
The second derivative of x(s) with respect to s, x’’(s) can be approximated by:
2.
Similarly, the approximation to the fourth derivative is given by: www.nr.no

The initial idea: Snakes
1.
Look at the x-component of the previous equations:
2.
After discretizing and using the previous approximations this can be written i matrix form.
www.nr.no
The initial idea: Snakes
1.
The matrix equation for the x component becomes:
A www.nr.no
The initial idea: Snakes
1.
For one particular x(s n
) the equation will be:
2.
Introducing
α
=w
2
,
β
=(-w this can be rewritten:
1
-4w
2
) and
γ
=(-2w
1
+6w
2
) www.nr.no
The initial idea: Snakes
1.
A more compact way of writing this is:
2.
3.
4.
The matrix A has special properties, it is a pentadiagonal matrix.
It is typically referred to as the stiffness matrix, it actually relates the positions of the points along the contour to each other.
So now all we need is a matrix inversion….?
www.nr.no

The initial idea: Snakes
1.
2.
4.
5.
6.
3.
Not quite that simple!
We look for a snake position where the internal forces are counterbalanced by the external forces.
If all we do is a matrix inversion in the previous eqaution we will find a curve v that locally is the best possible curve.
But we want a curve that is globally optimal.
The way we solve this is iterative.
We actually make the position of the snake a function of time , the snake can move.
www.nr.no
The initial idea: Snakes
1.
2.
What if we made x and y dependent on time (or iteration), that is we write x t and y t
.
With this change in mind, consider the following change in the previous equation:
3.
It is actually saying exactly the same thing as the previous equation, the wanted x t and y t is found when their transformation is counterbalanced by the external forces.
www.nr.no
The initial idea: Snakes
1.
Remember we had:
2.
3.
The physical interpretation of this is that we have found the wanted x and y vectors when the transformation of x and y is exactly counterbalanced by P x and P y
.
That is, when the internal forces are counterbalanced by the external forces.
www.nr.no
The initial idea: Snakes
1.
The beauty of this is that it allows us to write an iteration for xt and yt.
2.
Start by rewriting: www.nr.no

The initial idea: Snakes
1.
Start with:
2.
A slight rearrangement gives us:
3.
This is the basic iteration that lets us update x and y .
www.nr.no
The initial idea: Snakes
1.
2.
In order to obtain snakes that move in a physically more plausible way the nodes are typically given a mass and an initial speed.
A damping force is also often used (as if the snake was moving in some viscous medium).
www.nr.no
The initial idea: Snakes
1.
This was the initial idea of Kass and Witkin in their very important paper: Snakes: Active contour models , Michael Kass, Andrew Witkin and Demetri
Terzopoulos, International Journal of Computer
Vision, pages 321-331, 1988.
2.
This is an excellent paper that definitely deserves some study.
Appendix A: The Euler Lagrange
Equation
1.
The following slides deal with the Euler Lagrange equation and its derivation and are included primarily for the sake of completeness.
2.
The notation is kept as general as possible.
3.
Some of the blood-dripping details have been omitted for the sake of clarity. www.nr.no
Return to presentation www.nr.no

Appendix A: The Euler Lagrange
Equation
1.
2.
Leonard Euler is recognized as one of the greatest mathematicians of all times.
Read more about him at: http://www-history.mcs.standrews.ac.uk/Matheat icians/Euler.html
www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
The Euler Lagrange Equation arises within the framework of the calculus of variations.
2.
Unlike the case in static optimization problems, the problem in variational calculus is that of finding some function y(x) such that some integral:
3.
is either a minimum or a maximum.
www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
2.
Joseph-Louis Lagrange was an outstanding French mathematician.
Read more about him at: http://www-history.mcs.standrews.ac.uk/Mathem aticians/Lagrange.html
www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
2.
3.
4.
In this equation, f() is the functional, y’(x)=dy/dx, and x is the independent variable.
For y(x) to be an extremum it is required that any neighboring function is not an extremum.
We define a neighboring function of y by adding an arbitrary amount to y(x).
Parametrically this is written as y=y(
α
, x).
www.nr.no

Appendix A: The Euler Lagrange
Equation
1.
2.
Observe that for
α
=0, y=y(0,x)=y(x).
Furthermore:
3.
4.
where
η
(x) is some arbitrary function of x that has continuous first derivatives and vanishes at x x
2
(that is,
η
(x
1
)=0 and
η
(x
2
)=0).
1 and
Note that this is the way in which we define variations of our solution.
www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
For J(
α
) to be an extremum, we must have:
2.
3.
for all
η
(x)
Let’s look at: www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
Having defined variations to our solution, let’s look for conditions under which our solution is extremum by substituting y(
α
,x) in our functional:
Appendix A: The Euler Lagrange
Equation
1.
We have:
2.
Interchanging the order of integration and differentiation we get: www.nr.no
www.nr.no

Appendix A: The Euler Lagrange
Equation
1.
By the chain rule we get:
2.
But:
3.
and
Appendix A: The Euler Lagrange
Equation
1.
2.
This can be done by integration by parts.
Remember:
3.
Applying this to the previous integral we get: www.nr.no
www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
Substituting we get:
2.
Now observe that if we could reduce
∂ η
/
∂ x to g()
η
(x) then we could factor out
η
(x) and would be left with something that must be equal to zero.
www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
But we have defined
η
(x) to be zero at the limits of integration so the first term disappears.
2.
3.
From the second term it is possible to factor out
η
(x) which is what we wanted.
Substituting back into the original equation we get: www.nr.no

Appendix A: The Euler Lagrange
Equation
1.
We have found that:
2.
Thus we have:
Appendix A: The Euler Lagrange
Equation
1.
2.
A development very similar to what we have just gone through allows for the establishment of the
Euler Lagrange equation for cases where f is a function of y and its first and second derivatives.
This equation is: www.nr.no
www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
We have found that:
2.
3.
Since
η
(x) is arbitrary the remaining part of the integrand must be equal to zero as a necessary condition for J to be an extremum.
Thus:
4.
This is the Euler Lagrange Equation.
www.nr.no
Appendix A: The Euler Lagrange
Equation
1.
This equation is a central equation in mathematics and physics.
2.
It has many important uses, one of its ‘famous’ applications is to the solution of the
Brachistochrone problem, see: http://mathworld.wolfram.com/BrachistochroneProb lem.html
www.nr.no

Appendix B: Numerical approximations to derivatives.
1.
2.
3.
The following slides deal with the numerical approximation to derivatives and are included primarily for the sake of completeness.
The notation is kept as general as possible.
Some of the gory details have been omitted for the sake of clarity.
Return to presentation www.nr.no
Appendix B: Numerical approximations to derivatives.
1.
Consider a list of samples of f, let us call them f(n), n=….-2,-1,0,1,2…
2.
The definition of the derivative partly answers our question as to how we compute a numerical approximation to it.
www.nr.no
Appendix B: Numerical approximations to derivatives.
1.
The derivative of a function f(x), denoted f’(x) is defined by:
2.
3.
4.
It is a measure of how fast the function f changes with a change in x.
You all know that, for instance, the derivative f’(x) of f(x)=x 3 f’(x)=3x 2 .
is
But what if no analytical expression for f is known, what if we only have samples of f.
www.nr.no
Appendix B: Numerical approximations to derivatives.
1.
Remember the definition of f’(x):
2.
If we remove the limit we get:
3.
For h small, we would expect this to be fairly close to the true value for f’(x).
www.nr.no

Appendix B: Numerical approximations to derivatives.
1.
2.
The question is obviously how large the error is when we compute the derivative tin this way.
In order to analyse this, consider the Taylor expansion of f(x) around the point (x+h):
3.
Reorganizing this we find:
4.
Here, the term O(h) regroups terms including h and higher powers of h, it is a measure of the error we make.
www.nr.no
Appendix B: Numerical approximations to derivatives.
1.
Remember:
2.
Rearranging we get:
3.
Compare this with:
4.
This symmetrical way of computing f’(x) is better, the error we make will be much smaller.
www.nr.no
Appendix B: Numerical approximations to derivatives.
1.
A simple trick allows us to improve on this.
Remember first:
2.
It is also a simple matter to show that:
3.
The difference of these equations is:
Appendix B: Numerical approximations to derivatives.
1.
By similar reasoning, discrete approximations to higher order derivatives of f(x) can be found.
2.
This is a complex subject and this very simplistic exposition is in no way complete.
3.
The only intention is to give the reader a starting point for further study. www.nr.no
www.nr.no