Mean Shift Tracking
CS4243 Computer Vision and Pattern Recognition
Leow Wee Kheng
Department of Computer Science
School of Computing
National University of Singapore
(CS4243)
Mean Shift Tracking
1 / 1
Mean Shift
Mean Shift
Mean Shift [Che98, FH75, Sil86]
An algorithm that iteratively shifts a data point to the average of
data points in its neighborhood.
Similar to clustering.
Useful for clustering, mode seeking, probability density estimation,
tracking, etc.
(CS4243)
Mean Shift Tracking
2 / 1
Mean Shift
Consider a set S of n data points xi in d-D Euclidean space X.
Let K(x) denote a kernel function that indicates how much x
contributes to the estimation of the mean.
Then, the sample mean m at x with kernel K is given by
m(x) =
n
X
K(x − xi ) xi
i=1
n
X
i=1
(1)
K(x − xi )
The difference m(x) − x is called mean shift.
Mean shift algorithm: iteratively move date point to its mean.
In each iteration, x ← m(x).
The algorithm stops when m(x) = x.
(CS4243)
Mean Shift Tracking
3 / 1
Mean Shift
The sequence x, m(x), m(m(x)), . . . is called the trajectory of x.
If sample means are computed at multiple points, then at each
iteration, update is done simultaneously to all these points.
(CS4243)
Mean Shift Tracking
4 / 1
Mean Shift
Kernel
Kernel
Typically, kernel K is a function of kxk2 :
K(x) = k(kxk2 )
(2)
k is called the profile of K.
Properties of Profile:
1
k is nonnegative.
2
k is nonincreasing: k(x) ≥ k(y) if x < y.
3
k is piecewise continuous and
Z ∞
0
(CS4243)
k(x)dx < ∞
Mean Shift Tracking
(3)
5 / 1
Mean Shift
Kernel
Examples of kernels [Che98]:
Flat kernel:
K(x) =
½
1 if kxk ≤ 1
0 otherwise
(4)
Gaussian kernel:
K(x) = exp(−kxk2 )
(a) Flat kernel
(CS4243)
(5)
(b) Gaussian kernel
Mean Shift Tracking
6 / 1
Mean Shift
Density Estimation
Density Estimation
Kernel density estimation (Parzen window technique) is a popular
method for estimating probability density
[CRM00, CRM02, DH73].
For a set of n data points xi in d-D space, the kernel density
estimate with kernel K(x) (profile k(x)) and radius h is
f˜K (x) =
=
µ
¶
n
x − xi
1 X
K
nhd
h
i=1
ð
° !
n
° x − x i °2
1 X
°
k °
° h °
nhd
(6)
i=1
The quality of kernel density estimator is measured by the mean
square error between the actual density and the estimate.
(CS4243)
Mean Shift Tracking
7 / 1
Mean Shift
Density Estimation
The mean square error is minimized by the Epanechnikov kernel:

 1
(d + 2)(1 − kxk2 ) if kxk ≤ 1
(7)
KE (x) =
2Cd

0
otherwise
where Cd is the volume of the unit d-D sphere, with profile

 1
(d + 2)(1 − x) if 0 ≤ x ≤ 1
kE (x) =
2Cd

0
if x > 1
A more commonly used kernel is Gaussian
µ
¶
1
1
2
exp − kxk
K(x) = p
2
(2π)d
(8)
(9)
with profile
1
(CS4243)
µ
1
k(x) = p
exp − x
d
2
(2π)
Mean Shift Tracking
¶
(10)
8 / 1
Mean Shift
Density Estimation
Define another kernel G(x) = g(kxk2 ) such that
g(x) = −k 0 (x) = −
dk(x)
dx
(11)
Important Result
Mean shift with kernel G moves x along the direction of the gradient of
density estimate f˜ with kernel K.
Define density estimate with kernel K.
But, perform mean shift with kernel G.
Then, mean shift performs gradient ascent on density estimate.
(CS4243)
Mean Shift Tracking
9 / 1
Mean Shift
Density Estimation
Proof:
Define f˜ with kernel G
ð
° !
n
X
° x − x i °2
C
°
f˜G (x) ≡
g °
° h °
nhd
(12)
i=1
where C is a normalization constant.
Mean shift M with kernel G is
n
X
ð
° !
° x − x i °2
°
g °
° h ° xi
MG (x) ≡ i=1n ð
° ! −x
X
° x − x i °2
°
g °
° h °
(13)
i=1
(CS4243)
Mean Shift Tracking
10 / 1
Mean Shift
Density Estimation
Estimate of density gradient is the gradient of density estimate
˜ K (x) ≡ ∇f˜K (x)
∇f
=
=
ð
°2 !
n
°
°
2 X
0 ° x − xi °
(x − xi ) k °
d+2
nh
h °
i=1
ð
° !
n
° x − x i °2
2 X
°
(xi − x) g °
° h °
nhd+2
(14)
i=1
=
2 ˜
fG (x)MG (x)
Ch2
Then,
MG (x) =
˜ K (x)
Ch2 ∇f
2 f˜G (x)
(15)
Thus, MG is an estimate of the normalized gradient of fK .
So, can use mean shift (Eq. 13) to obtain estimate of fK .
(CS4243)
Mean Shift Tracking
11 / 1
Mean Shift Tracking
Mean Shift Tracking
Basic Ideas [CRM00]:
Model object using color probability density.
Track target object in video by matching color density.
Use mean shift to estimate color density and target location.
(CS4243)
Mean Shift Tracking
12 / 1
Mean Shift Tracking
Object Model
Object Model
Let xi , i = 1, . . . , n, denote pixel locations of model centered at 0.
Represent color distribution by discrete m-bin color histogram.
Let b(xi ) denote the color bin of the color at xi .
Assume size of model is normalized; so, kernel radius h = 1.
Then, the probability q of color u in the model is
qu = C
n
X
i=1
k(kxi k2 ) δ(b(xi ) − u)
C is the normalization constant
#−1
" n
X
2
k(kxi k )
C=
(16)
(17)
i=1
Kernel profile k weights contribution by distance to centroid.
(CS4243)
Mean Shift Tracking
13 / 1
Mean Shift Tracking
Object Model
δ is the Kronecker delta function
½
1 if a = 0
δ(a) =
0 otherwise
(18)
That is, contribute k(kxi k2 ) to qu if b(xi ) = u.
(CS4243)
Mean Shift Tracking
14 / 1
Mean Shift Tracking
Target Candidate
Target Candidate
Let yi , i = 1, . . . , nh , denote pixel locations of target centered at y.
Then, the probability p of color u in the target is
ð
° !
nh
X
° y − y i °2
°
δ(b(yi ) − u)
pu (y) = Ch
k °
° h °
(19)
i=1
Ch is the normalization constant
"n
ð
° !#−1
h
X
° y − y i °2
°
Ch =
k °
° h °
(20)
i=1
(CS4243)
Mean Shift Tracking
15 / 1
Mean Shift Tracking
Color Density Matching
Color Density Matching
Use Bhattacharyya coefficient ρ
ρ(p(y), q) =
m p
X
pu (y) qu
(21)
u=1
√
√
√
√
ρ is the cosine of vectors ( p1 , . . . , pm )> and ( q1 , . . . , qm )> .
Large ρ means good color match.
For each image frame, find y that maximizes ρ.
This y is the location of the target.
(CS4243)
Mean Shift Tracking
16 / 1
Mean Shift Tracking
Tracking Algorithm
Tracking Algorithm
Given {qu } of model and location y of target in previous frame:
1
Initialize location of target in current frame as y.
2
Compute {pu (y)} and ρ(p(y), q).
3
Apply mean shift: Compute new location z as
ð
° !
nh
X
° y − y i °2
°
g °
° h ° yi
ð
z = i=1
° !
nh
X
° y − y i °2
°
g °
° h °
(22)
i=1
4
5
6
Compute {pu (z)} and ρ(p(z), q).
While ρ(p(z), q) < ρ(p(y), q), do z ← 12 (y + z).
If kz − yk is small enough, stop. Else, set y ← z and goto (1).
(CS4243)
Mean Shift Tracking
17 / 1
Mean Shift Tracking
Tracking Algorithm
Step 3: In practice, a window of pixels yi is considered.
Size of window is related to h.
Step 5 is used to validate the target’s new location.
Can stop Step 5 if y and z round off to the same pixel.
Tests show that Step 5 is needed only 0.1% of the time.
Step 6: can stop algorithm if y and z round off to the same pixel.
To track object that changes size, varies radius h
(see [CRM00] for details).
(CS4243)
Mean Shift Tracking
18 / 1
Mean Shift Tracking
Tracking Algorithm
Example 1: Track football player no. 78 [CRM00].
(CS4243)
Mean Shift Tracking
19 / 1
Mean Shift Tracking
Tracking Algorithm
Example 2: Track a passenger in train station [CRM00].
(CS4243)
Mean Shift Tracking
20 / 1
Reference
Reference I
Y. Cheng.
Mean shift, mode seeking, and clustering.
IEEE Trans. on Pattern Analysis and Machine Intelligence,
17(8):790–799, 1998.
D. Comaniciu, V. Ramesh, and P. Meer.
Real-time tracking of non-rigid objects using mean shift.
In IEEE Proc. on Computer Vision and Pattern Recognition, pages
673–678, 2000.
D. Comaniciu, V. Ramesh, and P. Meer.
Mean shift: A robust approach towards feature space analysis.
IEEE Trans. on Pattern Analysis and Machine Intelligence,
24(5):603–619, 2002.
(CS4243)
Mean Shift Tracking
21 / 1
Reference
Reference II
R. O. Duda and P. E. Hart.
Pattern Classification and Scene Analysis.
Wiley, 1973.
K. Fukunaga and L. D. Hostetler.
The estimation of the gradient of a density function, with
applications in pattern recognition.
IEEE Trans. on Information Theory, 21:32–40, 1975.
B. W. Silverman.
Density Estimation for Statistics and Data Analysis.
Chapman and Hall, 1986.
(CS4243)
Mean Shift Tracking
22 / 1