IPAM-UCLA Tutorial
May 14-18, 2001
GBM in Image Processing,
Computer Vision, and
Computer Graphics
Guillermo Sapiro
Electrical and Computer Engineering
University of Minnesota
guille@ece.umn.edu
Supported by IPAM, NSF, ONR
GBM-IPAM
Guillermo Sapiro
1
GBM: Moving Curves
GBM-IPAM
Guillermo Sapiro
2
Basic curve evolution
 Planar curve:
C ( p ) : [0,1]  R2
 General flow:


C
 T  N
t
 General geometric flow:

C
 N
t
GBM-IPAM
Guillermo Sapiro
3
Mathematical morphology
 Classical theory, based on Minkowsky addition.
 The old and (probably wrong) way of doing geometric
image analysis.
 Has very important lessons to learn!!!!
 Basic definitions:
 A: Image in Euclidean space (R or Z)
 B: Structuring element (symmetric)
 Nothing else than Minkowsky addition
GBM-IPAM
Guillermo Sapiro
4
Mathematical morphology: Definitions
A  B:  {a  b, a  A, b  B}   Ab
bB
AB:  ( Ac  B) c   Ab
b B
A  B:  ( AB)  B 
A  B:  ( A  B)B
GBM-IPAM
Guillermo Sapiro
5

{ y : B y  A}
Ay
Mathematical morphology: Is it good or bad?
 Advantages:
 Nice mathematical properties (set theory)
 Extension to Lattices
 Disadvantages:
 Discrete Minkowsky addition does not look good, has to be
replaced by better ways of computing “discrete distances.”
 Major important concept: Level-sets
f : R N  R,
g: R N  {0,}
f  g  max { f }
support of g
 Commutes with thresholding (level-sets): Do binary on each
level sets or do gray-level on all the image => same result
 It is in certain sense a particular case of curve evolution
(before the lattices part)
GBM-IPAM
Guillermo Sapiro
6
Mathematical morphology via curve evolution
Convex structuring elements B:
A  ( B  C)  ( A  B)  C
A  (rB)  A  r1 B  r2 B...
Huygens principle
GBM-IPAM
Guillermo Sapiro
7
Mathematical morphology via curve evolution (cont.)
 General velocity:
  sup {r ( )  N}
 Examples:
N
  max{N x , N y }
B  disk
B  diamond
 | N x || N y |
B  square
 Nothing else than changing the metric (distance).
 Can be explained also based on dynamic programming
and time of arrival
 See Sapiro et al., Brocket-Maragos, Alvarez et al., Evans, Falcone
GBM-IPAM
Guillermo Sapiro
8
Planar differential geometry
 Euclidean invariant
parametrization
 Affine invariant
parametrization
 C( s )
 1
s
 C( s )  2 C( s )
,
 1
2
s
s
< Cs , Css >  0
Cs , Csss  0
Cs ^ Css
 Cs  Csss  0
 :  Css
GBM-IPAM
Guillermo Sapiro
  Css , Csss
9
Planar differential geometry (cont.)
 Curvature constant for circles
or straight lines (=0)
 Curvature defines curve up to
Euclidean motion
 Curvature constant for
ellipses (>0), hyperbolas (<0),
and parabolas (=0)
 Curvature defines curve up to
affine motion
 At least 4 points with dk/ds=0
 At least 6 points with dk/ds=0
 Defined for all curves
 Defined only for convex
curves: segment at inflection
points
GBM-IPAM
Guillermo Sapiro
10
Planar differential geometry (cont.)
d ( X , C( s )) :  < X  C( s ) , X  C( s ) > d ( X , C( s )) :  X  C( s ) , C ( s )
s
X
X
C(s)
C(s)
d
 < X  C , Cs >
s
d
 X  C( s ) , Css ( s )
s
 (  0)
 (  0)
X  C( s ) ^ Cs ( s )
X  C( s ) || Css ( s )
X  C( s ) || Css ( s )
Distance has a local extrema iff X is on the norma
GBM-IPAM
Guillermo Sapiro
11
3D Differential geometry
 Remember mean and Gaussian curvatures?
 Each regular surface has two principal curvatures. The
average is the mean curvature, the product the Gaussian.
These are also related to the tangential map, etc, etc. See
DoCarmo for details.
GBM-IPAM
Guillermo Sapiro
12
Riemannian geometry, Lie theory
 What about other non-Euclidean metrics?
 What about invariants to other (Lie) groups, e.g.,
projective?
 What about differential invariants? Semi-differential
invariants? Are there any general theories?
GBM-IPAM
Guillermo Sapiro
13
Smoothing by classical heat flow
C = C
t

 Linear
 Equivalent to Gaussian filtering
 Unique linear scale-space
 Non geometric
 Shrinks the shape
 Implementation problems
GBM-IPAM
Guillermo Sapiro
14
 xt 
y 
 t
=
 x pp 


y
 pp 
Invariant shape deformations
 Formulate shape deformations
 Geometric
 Invariant to camera transform
 The “best” possible
 Change only the desired features
 Motivation:
 Mathematics:


From static differential geometry to dynamic
Beautiful
 Computer vision and image processing:


Invariant shape segmentation and analysis
Image processing via image deformations
 Robotics:



GBM-IPAM
Guillermo Sapiro
Motion planning
Accurate geometric object detection and tracking
Robot manipulation and grasping
15
Basic planar differential geometry
 For every Lie group we will consider, exists and
invariant parametrization s, the group arc-length
 For every such a group exists and invariant
signature, the group curvature, k
Low curvature
High curvature
Negative curvature
GBM-IPAM
Guillermo Sapiro
16
What and why invariant
Camera
motion
Deformation
 Camera/object movement in the space
 Transformations description (for “flat” objects):
 Euclidean

Motion parallel to the camera and planar projection
 Affine

Planar projection
 Projective
GBM-IPAM
Guillermo Sapiro
17
Euclidean geometric heat flow
 Use the Euclidean arc-length: Cs = 1
 The deformation:

C
2 C
=
=

N
t
 s2
 Smoothly deforms to a circle (Gage-Hamilton, Grayson)
 Geometric smoothing
 Reduces length as fast as possible
GBM-IPAM
Guillermo Sapiro
18
Affine geometric heat flow (Sapiro-Tannenbaum)
 Use the affine arc-length:
 The flow:
Css = 
1/ 3



N  f ( , ) T
Css
C
= 
t
0
Ct = 
GBM-IPAM
Guillermo Sapiro
Cs  Css = 1
non - inflection
inflection
1/ 3

N
19
Affine geometric heat flow (cont.)
 Geometric smoothing (preserving area if desired)
 Total curvature decreases
 Maxima of curvature decreases
 Number of inflections decreases
 Smoothly deforms a shape into an ellipse
 Decreases area as fast as possible (in an affine form)
 Existence also for non-smooth curves
 Viscosity framework (Alvarez-Guichard-Morel-Lions)
 Polygons (Angenent-Sapiro-Tannenbaum)
 Applications:
 Curvature computation for shape recognition: reduce noise
(Morel et al.)
 Simplify curvature computation (Faugeras ‘95)
 Object recognition for robot manipulation (Cipolla ‘95)
GBM-IPAM
Guillermo Sapiro
20
General invariant flows
 Theorem: For every sub-group of the projective group the
most general invariant curve deformation has the form
C
2 C
=
f ( , s , ss ,...)
2
t
s
 Theorem: In general dimensions, the most general
invariant flow is given by
ut =
g
f ( curvatures)
E( g )
 u: graph locally representing the surface
 g: invariant metric
 E(g): variational derivative of g
 See Olver et al., Alvarez et al., Caselles-Sbert
GBM-IPAM
Guillermo Sapiro
21
General Geometric Framework For
Object Segmentation
GBM-IPAM
Guillermo Sapiro
22
Introduction
 Goal: Object detection
 Approach: Curve/surface deformation
 Geometry dependent regularization
 Image dependent velocity
 Characteristics:
 Unifies previously considered independent approaches
 Relates segmentation with anisotropic diffusion
 General:



Any topology
Any type of image data
Any dimension
 Holds formal results
GBM-IPAM
Guillermo Sapiro
23
Notation
 Deforming curve:
 Image:
GBM-IPAM
Guillermo Sapiro
C ( p ) : [0,1]  R2
I : [0,1] [0,1]  R
2
N
(R )
24
Basic active contours approach
 Terzopoulos et al., Cohen et al.
E(C )    C'( p) 2 dp    C''( p) 2 dp   I (C ) dp
 Drawbacks:
 Too many parameters
 Non-geometric
 Handling topology changes
GBM-IPAM
Guillermo Sapiro
25
Geodesic active contours
(Caselles-Kimmel-Sapiro)
E(C )    C'( p) 2 dp    C''( p) 2 dp   I (C ) dp
 Generalize image dependent energy
 Eliminate high order smoothness term
 Equal internal and external energies
E(C )   C'( p) 2 dp   g[| I (C( p))|] dp
 Maupertuis and Fermat principles of dynamical systems
E(C )   g[| I (C( s))|] ds
GBM-IPAM
Guillermo Sapiro
26
Geodesic computation
 Gradient-descent
E( C )   ds


C
 N
t



C
 g  N  g  N
t
E( C )   g[| I ( C( s))|] ds
 Level-sets (Osher-Sethian)
 C   N
t
GBM-IPAM
Guillermo Sapiro

    ||
 t
27
Further geometric interpretation
 The geodesic flow
 C  g  N  g  N
t
GBM-IPAM
Guillermo Sapiro
28

(  gN )
Model correctness
 Theorem: The deformation is independent of the level-sets
embedding function
 Theorem: There is a unique solution to the flow in the
viscosity framework
 Theorem: The curve converges to ideal objects when
present in the image
 Related work:





Kimia-Tannenbaum-Zucker
Caselles et al.
Malladi-Sethian-Vemuri
Kichenassamy at al.
Tek-Kimia, Whitacker
 New work:




Chan-Vese
Paragios-Deriche
Yezzi et al.
Faugeras et al.
GBM-IPAM
Guillermo Sapiro
29
Extensions
E(C )   g[| I (C( s))|] ds
 Gray-level values
 ds - length element (geodesics)
 Ordinary edge detector (gradient)
 Surfaces
 ds - area element (minimal surfaces)
 3D edge detector
 Vector-valued images (color, texture, medical, etc)
 ds - length element
 Vector-valued edge detector (vector geodesics)

Eigenvalues of the first fundamental form in Riemannian space
 Invariant detection (affine area geodesics)
 ds - affine length element (area related)
 Affine invariant edge detector
 Affine norm for “gradient descent”
GBM-IPAM
Guillermo Sapiro
30
Why color edges?
GBM-IPAM
Guillermo Sapiro
31
Notation
 Image
X : [0, N ]x[0, N ]  R
L*a*b*
 Texture: Gabor decomposition
GBM-IPAM
Guillermo Sapiro
32
N
Color edge computation
 X   X 2
i
i
 Given a metric (Euclidean)
 Compute first fundamental form
[ gij ]   X   X
i j
 Compute eigenvectors and eigenvalues
 Edge: maximal eigenvalue and its eigenvector
(,,,)
 Basic properties:
 Eigenvectors are orthonormal
 Minimal eigenvalue is not zero
GBM-IPAM
Guillermo Sapiro
33
GBM: Moving Images
GBM-IPAM
Guillermo Sapiro
34
Anisotropic diffusion
Isotropic vs. Anisotropic Smoothing
Isotropic
smoothing
GBM-IPAM
Guillermo Sapiro
Anisotropic
smoothing
35
Isotropic diffusion (Koenderink, Witkin)
 All “equivalent:”
 Gaussian filtering of the image
 Heat flow
   
t
 Minimize the L2 norm
 
GBM-IPAM
Guillermo Sapiro
36
2

Isotropic diffusion: Good things
 Gaussian filtering if and only if
 Linear
 Shift-invariant
 No creation of zero crossings
 Gaussian filtering if and only if
 Linear
 Shift-invariant
 Semi-group property
 Scale-invariant (dimensionless)
 Unique linear filter that defines a scale-space: Do not
creates information at coarser scales
 Where everything started (Koenderink, Witkin)
GBM-IPAM
Guillermo Sapiro
37
Isotropic diffusion: Bad things and possible solutions
 Non-geometric
 Problems with implementations
 Who said linear? Replace heat flow by “parabolic” PDE’s
(Hummel’s original idea)
 Why parabolic? Because of the maximum principle.
GBM-IPAM
Guillermo Sapiro
38
Perona-Malik anisotropic diffusion
 Replace the L2 by a different norm (e.g., L1, Rudin-OsherFatemi; Lorentzian, Black et. al.; etc)
 h(  )
GBM-IPAM
Guillermo Sapiro



t
39

div h'( I )

I
I




Selection of “stopping term” h
 How do we select h?
 h=x*x => L2 => linear => Isotropic diffusion
 h=x
=> L1 (Rudin-Osher-Fatemi)
(

)

GBM-IPAM
Guillermo Sapiro




t
40

Robust anisotropic diffusion
 General theory for selection “h”, based on the theory of
influence functions in robust statistics
 Edges should be considered outliers: At certain point, h’,
the influence, should be zero.
L
GBM-IPAM
Guillermo Sapiro
41
Directional diffusion
 Diffuse in the direction perpendicular to the edges (Avarez
et al.)
 
t
  = 

^
GBM-IPAM
Guillermo Sapiro
42
From active contours to anisotropic diffusion
 Replace embedding function in level-sets formulation
by image itself

 g( I )    g (I)  
t
I
 g( I )  I  g( I )  I
t
Shock-filters
(Osher-Rudin)
Anisotropic diffusion
(Alvarez et al.)
GBM-IPAM
Guillermo Sapiro
43
Relation with Perona-Malik anisotropic diffusion

 g( I )    g (I)  
t



t
 h(  )


div g( I )




t
I
I





div h'( I )

I
I
Total variation, Robust estimation Anisotropic diffusion
GBM-IPAM
Guillermo Sapiro
44




Concluding remarks
Terzopoulos snakes
Geometric interpretation
Dynamical systems
Level-sets
Terms
elimination
Curve evolution
active contours
Geodesic active
contours
Use image as embedding
Geometric diffusion
Shock-filters
Add
Self-snakes
Mumford-Shah
Divide by gradient
Perona-Malik flow
Variational interpretation
Total Variation
GBM-IPAM
Guillermo Sapiro
Robust Estimation
45
Anisotropic Diffusion
of the Posterior
GBM-IPAM
Guillermo Sapiro
46
ADP in MRI
Review: MAP Estimation
 3 classes: sulcus, gray matter, white matter
 Prior probability: Pr(class=C)
 Posterior probability: Pr(class=C | data)
 MAP: Choose class C that maximizes posterior:
C* = arg max Pr(class=C | data)
C
 Bayes’ Rule:
Pr(class=C | data) = Pr(data | class=C).Pr(class=C)
Pr(data)
 What is our prior, Pr(class=C)?
GBM-IPAM
Guillermo Sapiro
47
ADP: Common Techniques
MAP Estimation: Uniform Prior
Classification
GBM-IPAM
Guillermo Sapiro
48
ADP: Results
Anisotropic smoothing of posterior (Teo-Sapiro-Wandell)
Posterior
GBM-IPAM
Guillermo Sapiro
Smoothed
Posterior
49
Classification
ADP: Comparisons
Comparison with manual segmentation
Automatic
(2 min)
MR Image
GBM-IPAM
Guillermo Sapiro
Manual
(18 hours)
50
SAR segmentation via vector probability processing
With A. Pardo (see also Haker-Sapiro-Tannenbaum)
GBM-IPAM
Guillermo Sapiro
51
Anisotropic Diffusion in Vector Space
GBM-IPAM
Guillermo Sapiro
52
Goal and approach
(Ringach-Sapiro)
 Goal:
 Enhancement of vector valued data
 Extend classical theories of scalar PDE’s in image
processing
 Approach:
 Work in vector space
 Compute vector edges
 Anisotropic diffusion
 Important: Works for any vector data
 See also: Cumani, Di Zenzo, Chambolle
GBM-IPAM
Guillermo Sapiro
53
Notation
 Image
X : [0, N ]x[0, N ]  R
L*a*b*
 Texture: Gabor decomposition
GBM-IPAM
Guillermo Sapiro
54
N
Color edge computation
 X   X 2
i
i
 Given a metric (Euclidean)
 Compute first fundamental form
[ gij ]   X   X
i j
 Compute eigenvectors and eigenvalues
 Edge: maximal eigenvalue and its eigenvector
(,,,)
 Basic properties:
 Eigenvectors are orthonormal
 Minimal eigenvalue is not zero
GBM-IPAM
Guillermo Sapiro
55
Color anisotropic diffusion
 Direction: Minimal change
 Strength: g( ,  )
()

 X  g( , )  X
_

t
 2
2
GBM-IPAM
Guillermo Sapiro
56
Level lines for vectorial images (Chung-Sapiro)
Vector and scalar representation
sharing level-lines
GBM-IPAM
Guillermo Sapiro
57
Contrast Enhancement
(Sapiro-Caselles, and Caselles-Lisani-Morel-Sapiro)
 Contrast enhancement via image deformations
 Approach: Histogram modification
 I ( x, y)  I ( x, y)  (# pixels of value  I ( x, y) )
t
U(I) 
1
2


2
[
I
(
x
)

1
/
2
]
d
x


1
4


 
[
I
(
x
)

I
(
z
)]
d
x
dz

 Characteristics:





GBM-IPAM
Guillermo Sapiro
Simultaneous contrast enhancement and denoising
First explanation of histogram modification in image domain
Extended to local
First semi-global partial differential equation in image processing
Formal existence results
58
GBM: Beyond the flat manifolds
GBM-IPAM
Guillermo Sapiro
59
The main problem and our goal
(Tang-Sapiro-Caselles)
 Goal: Enhancement and analysis of directional data (and
data on non-flat manifolds)
 Problem: Directions are unit vectors:
 Regular images vs Directions
2
I: R  R
N
vs
2
I: R  S
N 1
 Applications:
 Optical flow, Gradients
 Vector data (normalized)
 Color image enhancement
 Surface normals and principal directions
 Flows in general manifolds
GBM-IPAM
Guillermo Sapiro
60
Average
S
R
1
2
GBM-IPAM
Guillermo Sapiro
61
Most popular previous approaches
 Work with angles: Operations on the sphere
 Average, median, etc
 Statistics of directional data, Mardia
 “Orientation Diffusion,” Perona (1998)
 Tensor diffusion
 Weickert, Granlund-Knuttson
 See also Chan-Shen
GBM-IPAM
Guillermo Sapiro
62
Anisotropic Diffusion
Isotropic
(Heat equation)
 I ( x, y,t) I
t
GBM-IPAM
Guillermo Sapiro
Anisotropic
 I ( x, y,t)  div( g(|I |)I )
t
63
What have we learned from images?
Robust Estimation:minI  (|I |)d


Robust function
Gradient Descent:  I ( x, y,t )  div ' I
t


'
g :
















|I |
Influence function
(defines outliers)
|I |
 I ( x, y,t)  div( g(|I |)I )
Anisotropic Diffusion:
t
GBM-IPAM
Guillermo Sapiro
64
Anisotropic Diffusion
Isotropic
(Heat equation)
 I ( x, y,t) I
t
GBM-IPAM
Guillermo Sapiro
Anisotropic
 I ( x, y,t)  div( g(|I |)I )
t
65
Back to Directions: Basic Idea
 Use the theory of harmonic maps
 Find a map I from two manifolds (M,g) and (N,h) such that
min I : M  N

M
I
p
dvol M

 In particular, liquid crystals:
min
GBM-IPAM
Guillermo Sapiro
2
I :R S
n 1
I

66
p
dx dy
The Gradient-Descent Equations
General (p=2):
I
  M I  AN ( I ) <  M I ,  M I >
t
Liquid crystals:
I
 div ( I
t
GBM-IPAM
Guillermo Sapiro
p2
I )  I I
67
p
A Few Theoretical Results (over hundreds relevant)
 For 2D unit vectors (n=1), and p=2, a unique solution exists
and singularities are isolated points (if they exists at all).
For smooth data, singularities do not occur.
 Singularities occur for 3D unit vectors (p=2).
 Singularities well characterized for 1<p<=2.
 Energy well characterized for 1<p<=2.
 No singularities for manifolds with non-positive curvature.
GBM-IPAM
Guillermo Sapiro
68
Intermezzo: Visualizing Directions
 Arrows
 Color Map
 Line Integral Convolution

GBM-IPAM
Guillermo Sapiro

69
Examples (Isotropic)
GBM-IPAM
Guillermo Sapiro
70
3D vector (Isotropic)
GBM-IPAM
Guillermo Sapiro
71
Denoising
GBM-IPAM
Guillermo Sapiro
72
GBM-IPAM
Guillermo Sapiro
73
GBM-IPAM
Guillermo Sapiro
74
Optical flow
GBM-IPAM
Guillermo Sapiro
75
Optical flow (cont.)
GBM-IPAM
Guillermo Sapiro
76
Gradient
GBM-IPAM
Guillermo Sapiro
77
Gradient (cont.)
GBM-IPAM
Guillermo Sapiro
78
Color Image Enhancement
GBM-IPAM
Guillermo Sapiro
79
Color image enhancement (cont.)
GBM-IPAM
Guillermo Sapiro
80
Color image enhancement (cont.)
GBM-IPAM
Guillermo Sapiro
81
Vector probability diffusion (with Alvaro Pardo)
 Perform diffusion on the hyperplane representing
probabilities
min I :R 2 Hn
I
p
dx dy

 I
  M I  AN ( I ) <  M I ,  M I >
t
 I
p2
 div ( I
I )
t
GBM-IPAM
Guillermo Sapiro
82
Vector probability diffusion (cont.)
 The numerical implementation also stays on the
hyperplane
 The numerical implementation also holds a maximum and
minimum principle
GBM-IPAM
Guillermo Sapiro
83
Vector probability diffusion (cont.)
 Diffuse posterior probabilities (following Teo-Sapiro-Wandell
and Haker-Sapiro-Tannenbaum)
GBM-IPAM
Guillermo Sapiro
84
Current Results and Future Research
 Novel framework for analysis of directional data
 Isotropic and anisotropic
 Works in any dimension
 Supported by theoretical results on existence, uniqueness,
singularity classification
 See also Sochen-Kimmel-Malladi IEEE-IP ‘98,
Shen UCLA ‘99, Osher-Vese UCLA ‘00.
GBM-IPAM
Guillermo Sapiro
85
Chan-