ROBUST VIDEO STABILIZATION
BASED ON PARTICLE FILTER
TRACKING
OF PROJECTED CAMERA MOTION
IEEE TRANSACTIONS ON CIRCUITS AND
SYSTEMS FOR VIDEO TECHNOLOGY, 2009
Keywords:
Bootstrap filtering,
motion analysis,
Particle filtering,
video stabilization
P48981023
冷玉琦 Yu-Chi Leng
2011/01/16
ABSTRACT

We propose a novel technique for video stabilization
based on the particle filtering framework.

Extend the traditional use of particle filters in object
tracking to tracking of the projected affine model of the
camera motions.

The correspondence between scale-invariant feature
transform points is used to obtain a crude estimate of the
projected camera motion.
INTRODUCTION


Video cameras mounted on handheld devices and mobile
platforms have become increasingly popular in the
consumer market over the past few years due to a
dramatic decrease in the cost of such devices.
Stabilization methods exploit the fact that
camera motion causes the affine transform of the
frames, which can be inverted to obtain stable
frames.
THEORETICAL FOUNDATIONS

A. Camera Model
攝影機移動前後向量關係
[ x1
y1
z1 ]  R 3 x 3  [ x 0
[u 0
v0
] 
[u1
v1
] 
T

T
z0
u1 
 R 11

s
 

 v1 
 R 21
T

z1
 [ x0
y0
 [ x1
y1
z1 ]
1  R11  R12
2
T
z0 ]
R 12   u 0   t x 
  v   t 
R 22   0   y 
R11 R 21  R12 R 22 
z 0 ]  T3 x 1
y0
2
T
T
6個自由度!
1  R 21  R 22  0
2
2
THEORETICAL FOUNDATIONS

B. Particle Filtering Estimation
狀態向量： x  [ s , R , R , R , t , t ]
事後機率密度函數：p ( x z )
T
k
k
11 k
12 k
k
N
21 k
xk
yk
1: k
p ( x k z 1:k )   w k  ( x k  x k )
i
i
i 1
w k  p ( x k z 1:k ) q ( x k z 1:k )
i
估算當前狀態：
xˆ k  E x k    x k p ( x k z 1:k ) d x k   x k  w k  ( x k  x k )d x k   w k x k
N
i 1
重要性密度： x
i
k
~ q(x k , 1 )
i
i
N
i 1
i
i
THEORETICAL FOUNDATIONS
誤差向量
 k  xˆ k  x k
ek  xk  xk
協方差矩陣
Cov (  k ,  k )
Cov (  k ,  k ) 
Cov ( e k , e k )
1
(  1  2 ) c k
N
Cov (  k ,  k )  Cov ( e k , e k )   2
1
N

2
1j
  2 j c k   2 j
N  max
2
j  1 ... M
2
c 
k
2
1j
 2j  2j Nk
2
2
O
VIDEO STABILIZATION

A. Importance Density Using Scale-Invariant
Features
影像間的特徵點將使用SIFT獲得。
使用特徵追蹤來得到中值向量： x  [ s , R , R , R , t , t ]
k
u k

 ..
vk 
 u k 1

s


.. 
 ..
 s k R11 k
1 
  s k R 21 k
1
 tx

v k 1
..
k
x k ~ qG (x k ,  1 ) 
i
k
11 k
12 k
21 k
xk
yk
s k R12 k 

s k R 22 k

t y 
k
T
 1 i

1
i




exp

x

x

x

x
k
1
k
k
6
 2 k

( 2 )  1
1
有效的減少粒子數至30個並有同等或超越300個粒子
所得的品質。
T
VIDEO STABILIZATION

B. Particle Filtering for Global Motion
Estimation Between Successive Frames
-權重值依粒子有多靠近真實的狀態來給定。
-選擇均方差(mean square error, MSE)及特徵距離做為
相似度的兩個量測量。
PMSE 
i
 Mi 
exp  
2 
2


M 
2
1
2 σ M
i
wk 
i
P feature 
i
 D 
exp   i 2 
2 σ F
 2 F 
1
2
i
PMSE P feature
i
i
 i 1 PMSE P feature
N
VIDEO STABILIZATION
利用離散權重近似真實狀態，得到仿射動態參數：
Rˆ 11 k
xˆ k  [ sˆ k
Rˆ 12 k
Rˆ 21 k
tˆxk
T
tˆyk ]
尺度因子、旋轉矩陣和位移與參考影像的關係：
s k  s k 1  sˆ
A
Tk
A
A
 Rˆ 11 k
 sˆ k 
ˆ
 R 21 k
A
Rk
 Rˆ 11 k
 R k 1 
ˆ
 R 21 k
A
 tˆx 
Rˆ 12 k  A
 T k 1   ˆ 
ˆ
R 22 k 
t y 
k
k
Rˆ 12 k 

Rˆ 22 k 
VIDEO STABILIZATION

C. Intentional Motion Estimation and
Motion Compensation
沿著x方向的平移和平移速度 將可表示為：
 Tx k   1
 v  
Tx k   0
1 Tx k 1   0 
 Tx v    
1  k 1   n 
補償非預期動態：
D
 u k 

u s  s k
D
A 1
A
D

   A R k  R k      T k   T k
vs  sk
  vk 

[ u k ,v k ] 不穩定影像像素點
[ u s ,v s ] 補償後穩定影像像素點
VIDEO STABILIZATION

Algorithm：
EXPERIMENTAL RESULTS
EXPERIMENTAL RESULTS
EXPERIMENTAL RESULTS
EXPERIMENTAL RESULTS
EXPERIMENTAL RESULTS
EXPERIMENTAL RESULTS
CONCLUSION



In this paper, we presented a novel approach for
robust video stabilization based on particle filter
estimation of projected camera motion.
An efficient implementation of particle filters for
global motion estimation has been proposed
based on carefully designed importance sampling.
We demonstrated experimentally that the
proposed particle filtering scheme can be used to
obtain an efficient and accurate motion
estimation in video sequences.