‫گروه بینایی ماشین و پردازش تصویر‬
Machine Vision and Image Processing Group
Mean Shift ; Theory and Applications
Presented by:
Reza Hemati
1
89 ‫دی‬
December 2010
Mean Shift
Theory and Applications
Yaron Ukrainitz & Bernard Sarel
2
Agenda
• Mean Shift Theory
• What is Mean Shift ?
• Density Estimation Methods
• Deriving the Mean Shift
• Mean shift properties
• Applications
• Clustering
• Discontinuity Preserving Smoothing
•Segmentation
• Object Tracking
• Object Contour Detection
3
Mean Shift Theory
4
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
5
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
6
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
7
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
8
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
9
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
10
Intuitive Description
Region of
interest
Center of
mass
Objective : Find the densest region
Distribution of identical billiard balls
11
What is Mean Shift ?
A tool for:
Finding modes in a set of data samples, manifesting an
underlying probability density function (PDF) in RN
PDF in feature space
• Color space
Non-parametric
• Scale spaceDensity Estimation
• Actually any feature space you can conceive
Discrete PDF Representation
•…
Data
Non-parametric
Density GRADIENT Estimation
(Mean Shift)
PDF Analysis
12
Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Data point density
implies PDF value !
Assumed Underlying PDF
Real Data Samples
13
Non-Parametric Density Estimation
Assumed Underlying PDF
Real Data Samples
14
Non-Parametric
Density Estimation
?
Assumed Underlying PDF
Real Data Samples
15
Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
PDF(x) =
c e

( x-μi )2
2 i 2
i
i
Estimate
Assumed Underlying PDF
Real Data Samples
16
Kernel Density Estimation
Parzen Windows - Function Forms
1 n
P ( x)   K ( x - x i )
n i 1
A function of some finite number of data points
x1…xn
Data
In practice one uses the forms:
d
K (x)  c k ( xi )
or
i 1
Same function on each dimension
K (x)  ck  x

Function of vector length only
18
Kernel Density Estimation
Various Kernels
1 n
P ( x)   K ( x - x i )
n i 1
Examples:
A function of some finite number of data points
x1…xn


 c 1 x
• Epanechnikov Kernel K E (x)  

 0
2
Data

x 1
otherwise
• Uniform Kernel
c
x 1
KU (x)  
 0 otherwise
• Normal Kernel
 1 2
K N (x)  c  exp   x 
 2

19
Kernel Density Estimation
Gradient
1 n
 P ( x)    K ( x - x i )
n i 1
Using the
Kernel form:
We get :
Give up estimating the PDF !
Estimate ONLY the gradient
 x - xi
K (x - xi )  ck 
 h

2



Size of window
 n

x
g
i i

c n
c n
 
i 1
P(x)   ki    gi   n
 x
n i 1
n  i 1  

gi

 i 1

20
g(x)  k (x)
Computing
Kernel Density
The Estimation
Mean Shift
Gradient
 n

x
g
i i

c n
c n
 
i 1
P(x)   ki    gi   n
 x
n i 1
n  i 1  

gi

 i 1

21
g(x)  k (x)
Computing The Mean Shift
 n

x
g
i i

c n
c n
 
i 1
P(x)   ki    gi   n
 x
n i 1
n  i 1  

gi

 i 1

Yet another Kernel
density estimation !
Simple Mean Shift procedure:
• Compute mean shift vector
 n

 x - xi 2 
  xi g 




h
 i 1

  x
m ( x)  

2
n


x
x


i
g





 h 

 i 1 

•Translate the Kernel window by m(x)
22
g(x)  k (x)
Mean Shift Mode Detection
What happens if we
reach a saddle point
?
Perturb the mode position
and check if we return back
Updated Mean Shift Procedure:
• Find all modes using the Simple Mean Shift Procedure
• Prune modes by perturbing them (find saddle points and plateaus)
• Prune nearby – take highest mode in the window
23
Mean Shift Properties
• Automatic convergence speed – the mean shift
vector size depends on the gradient itself.
• Near maxima, the steps are small and refined
Adaptive
Gradient
Ascent
• For Uniform Kernel (
), convergence is achieved in
a finite number of steps
• Normal Kernel (
) exhibits a smooth trajectory, but
is slower than Uniform Kernel (
).
24
Real Modality Analysis
Tessellate the space
with windows
Run the procedure in parallel
25
Real Modality Analysis
26
The blue data points were traversed by the windows towards the mode
Real Modality Analysis
An example
Window tracks signify the steepest ascent directions
27
Mean Shift Strengths & Weaknesses
Strengths :
Weaknesses :
• Application independent tool
• The window size (bandwidth
selection) is not trivial
• Suitable for real data analysis
• Does not assume any prior shape
(e.g. elliptical) on data clusters
• Can handle arbitrary feature
spaces
• Inappropriate window size can
cause modes to be merged,
or generate additional “shallow”
modes  Use adaptive window
size
• Only ONE parameter to choose
• h (window size) has a physical
meaning, unlike K-Means
28
Mean Shift Applications
29
Clustering
Cluster : All data points in the attraction basin of a mode
Attraction basin : the region for which all trajectories lead to the same mode
30
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer
Clustering
Synthetic Examples
Simple Modal Structures
Complex Modal Structures
31
Clustering
Feature space:
L*u*v representation
Modes found
Real Example
Modes after
pruning
Initial window
centers
Final clusters
32
Clustering
Real Example
L*u*v space representation
33
Clustering
Real Example
2D (L*u)
space
representation
Final clusters
34
Discontinuity Preserving Smoothing
Feature space : Joint domain = spatial coordinates + color space
 xs   xr 
K (x)  C  k s 
  kr 

h
h
s
r


 
Meaning : treat the image as data points in the spatial and gray level domain
Image Data
(slice)
Mean Shift
vectors
Smoothing
result
35
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer
Discontinuity Preserving Smoothing
Flat regions induce the modes !
z
37
y
Discontinuity Preserving Smoothing
The effect of
window size
in spatial and
range spaces
38
Discontinuity Preserving Smoothing
Example
39
Discontinuity Preserving Smoothing
Example
40
Segmentation
Segment = Cluster,
or Cluster of Clusters
Algorithm:
• Run Filtering (discontinuity preserving smoothing)
• Cluster the clusters which are closer than window size
Image Data
(slice)
Mean Shift
vectors
Smoothing
result
Segmentation
result
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer41
http://www.caip.rutgers.edu/~comanici
Segmentation
Example
…when feature space is only
gray levels…
42
Segmentation
Example
43
Segmentation
Example
44
Segmentation
Example
45
Segmentation
Example
46
Segmentation
Example
47
Segmentation
Example
48
Non-Rigid Object Tracking
…
…
49
Mean-Shift Object Tracking
General Framework: Target Representation
Choose a
reference
model in the
current frame
…
Current
frame
Choose a
feature space
…
Represent the
model in the
chosen feature
space
51
Mean-Shift Object Tracking
General Framework: Target Localization
Start from the
position of the
model in the
current frame
Search in the
model’s
neighborhood
in next frame
Find best
candidate by
maximizing a
similarity func.
Repeat the
same process
in the next pair
of frames
…
Model
Candidate
Current
frame
…
52
Mean-Shift Object Tracking
Target Representation
Choose a
reference
target model
Represent the
model by its
PDF in the
feature space
Choose a
feature space
0.35
Quantized
Color Space
Probability
0.3
0.25
0.2
0.15
0.1
0.05
0
1
2
3
.
.
.
m
color
53
Kernel Based Object Tracking, by Comaniniu, Ramesh, Meer
Mean-Shift Object Tracking
Target Model
Target Candidate
(centered at 0)
(centered at y)
0.35
0.3
0.3
0.25
0.25
Probability
Probability
PDF Representation
0.2
0.15
0.1
0.2
0.15
0.1
0.05
0.05
0
0
1
2
3
.
.
.
m
1
2
color
q  qu u 1..m
3
.
.
.
m
color
m
q
u 1
u
1
Similarity f y  f  q , p y 
    
Function:
p  y    pu  y u 1..m
m
p
u 1
u
1
54
Mean-Shift Object Tracking
Finding the PDF of the target model
xi i1..n
candidate
model
Target pixel locations
y
0
k ( x)
A differentiable, isotropic, convex, monotonically decreasing kernel
b( x )
The color bin index (1..m) of pixel x
• Peripheral pixels are affected by occlusion and background interference
Probability of feature u in model
 

k xi
b ( xi ) u
 y  xi
pu  y   Ch  k 
 h
b ( xi ) u 
2
0.3
0.3
0.25
0.25
0.2
Pixel weight
0.15
0.1
Normalization
factor
Probability
Normalization
factor
Probability
qu  C
Probability of feature u in candidate
2



0.2
0.15
Pixel weight
0.1
0.05
0.05
0
0
1
2
3
.
color
.
.
m
1
2
3
.
color
.
.
m
56
Mean-Shift Object Tracking
Similarity Function
Target model:
q   q1,
, qm 
Target candidate:
p  y    p1  y  ,
Similarity function:
f  y   f  p  y  , q   ?
, pm  y 
The Bhattacharyya Coefficient
q 

q1 ,
, qm

p1  y  ,
p  y  

q
1
, pm  y 

y
1
m
p  y  q
f  y   cos  y 
  pu  y  qu
p  y   q u 1
p  y 
T
57
Mean-Shift Object Tracking
Target Localization Algorithm
Start from the
position of the
model in the
current frame
q
Search in the
model’s
neighborhood
in next frame
p  y
Find best
candidate by
maximizing a
similarity func.
f  p  y  , q 
58
Mean-Shift Object Tracking
Approximating the Similarity Function
m
f  y    pu  y  qu
u 1
Linear
approx.
(around y0)
Model location:
y0
Candidate location:
1 m
1 m
f  y    pu  y0  qu   pu  y 
2 u 1
2 u 1
y
qu
pu  y0 
 y  xi
pu  y   Ch  k 
 h
b ( xi ) u 
Independent of
y
Ch
2
 y  xi
wi k 

 h
i 1

n
2



2



Density
estimate!
(as a function
of y)
59
Mean-Shift Object Tracking
Maximizing the Similarity Function
The mode of
Ch
2
 y  xi
wi k 

 h
i 1

n
2

 = sought maximum

Important Assumption:
The target
representation
provides sufficient
discrimination
One mode in
the searched
neighborhood
60
Mean-Shift Object Tracking
Applying Mean-Shift
The mode of
Ch
2
 y  xi
wi k 

 h
i 1

2
n

 = sought maximum

 y0  xi 2 
xi g 



h
i 1


y1 
2
n
 y0  xi 
g



h
i 1


n
n
 y  xi
Original
Find mode of c k 
Mean-Shift:
i 1
 h
2



using
 y0  xi 2 
xi wi g 


2

h

i 1
y  xi


y

using

1
n
 y0  xi 2 
h

wi g 



h
i 1


61
n

Extended
Find mode of c wi k 
Mean-Shift:
i 1

n
Mean-Shift Object Tracking
About Kernels and Profiles
A special class of radially
symmetric kernels:
 
K  x   ck x
2
The profile of
kernel K
k   x   g  x 
 y0  xi 2 
xi wi g 


2

h

i 1
y  xi


y

using

1
n
 y0  xi 2 
h

wi g 



h
i 1


62
n

Extended
Find mode of c wi k 
Mean-Shift:
i 1

n
Mean-Shift Object Tracking
Choosing the Kernel
A special class of radially
symmetric kernels:
 
K  x   ck x
2
Epanechnikov kernel:
Uniform kernel:
1  x if x  1 
k  x  

0
otherwise


1 if x  1 
g  x   k  x   

0
otherwise


 y0  xi 2 
xi wi g 



h
i 1


y1 
n
 y0  xi 2 
wi g 



h
i 1


n
n
y1 
xw
i 1
n
i
i
w
i 1
i
63
64