Robust Video Stabilization Based on Particle Filter Tracking of
Projected Camera Motion
(IEEE 2009)
Junlan Yang
University of Illinois,Chicago
1

• [1]A tutorial on particle filters for online nonlinear non-
Gaussian Bayesian tracking
• [4]probabilistic video stabilization using kalman filtering and mosaicking
• [5]Fast electronic digital image stabilization for off-road navigation
• [18]condensation conditional density propagation for visual tracking
2

 Introduction
 Camera Model
 Particle Filtering Estimation
 Complete System of Video Stabilization
 Simulation and Results
 Conclusion
3

• Video Stabilization
– Camera motion estimation
• Particle filter
– Tracking projected affine model of camera motion
• SIFT algorithm ( 范博凱 )
– Detect feature points in both images
• Removing undesired (unintended) motion
– Kalman filter
4

 Introduction
 Camera Model
 Particle Filtering Estimation
 Complete System of Video Stabilization
 Simulation and Results
 Conclusion
5

P
(x
1
,y
1
,z
1
)
Z
P
(x
0
,y
0
,z
0
)
Z
Y at time t
1
X
Camera
Motion
Y at time t
0
X
Camera
6

• Related of two vectors where R
3 3
,T are the transform of camera's
3-D rotation and translation, repectively
7

• Projection of P in time t0 and t1
Z
X
Y
λ
(u
0
,v
0
,λ)
(x
0
,y
0
,z
0
)
8

• Rewriting the related of two projected vectors
• 2-D affine model t where y
we sR
23
λ define s z
0
/z
1
, t x

( λ /z
1
)T y sR
13
λ 
( λ /z
1
)T x
9

R is orthonrmal matrix
Global motion estimation is to determine the six parameters for every successive frame
10

Why do she use 2-D affine model to represent camera motion?
 A pure 2-D model
 2-D translation vector and one rotation angle
 3-D model
 Giant complexity
11

 Introduction
 Camera Model
 Particle Filtering Estimation
 Complete System of Video Stabilization
 Simulation and Results
 Conclusion
12

• Markov discrete-time state-space model state vector at time k x k
[ s k
, t xk
, t yk
, R
11 k
, R
12 k
, R
21 k
]
T observations z , and the posterior density is p(x |z ) k 1:k
 i k

 i w ,i = 1,...,N ,where N is the number of particles k

and k is the time step
13

x k i
~ q (.)
...
3 2 1 N i = k i k are random vectors drawn from a proposal q() ,and the q() refered as an importance density
14

Estimation of current state
As N
 
, approximat ion converge to true posterior density p(x k
| z
1 :k
) in mean square sense and convergenc e rate
1
N
15

• Traditionally – prior density p(x k
| x k 1
)
• This paper takes into account the current observation z k
. The proposed important density x current observation z k k
• Why do she use the particle filtering estimation ?
16

Advantage of particle filtering estimation
• With Low error variance
• Proof : In large particle numbers condition, the
ˆx k x k x i k
~ q( x k
,

1
) , where

1 is set to be diagonal and x k is obtained from feature based motion estimation .
We consider a fine estimate that x k is an unbiased estimation of true state x k
with diagonal covariance matrix

2
17

ε k e k xˆ k x k

 x k
, covariance x k
, covariance error as Cov(ε k
, error as Cov(e k
ε k
)
, e k
)
 
2 given unbiasedne ss assumption
In order to simplify t he prove , she set the
true state x k as in the origin
ε k xˆ k and e k x k
.
Both of them have zero mean
18

where
19

x i k
~ G( x k
,

1
)
E[x i k x i k
T
E[x i k x i k
T
| x k
]

E[x i k
| x k
]E[x i k
T
| x k
]
 
1
| x k
]
 
1
 x k x k
T
20

w i k
 π i k
/ i
N 

1
π i k
, where π i k are the likelihood computed for i.i.d
particles, and regarded as i.i.d
random variables with mean m
π and variance
σ 2
π
, varing with k
Strong law of large number
Denote c k
(m
2
π
 σ 2
π
)/m
2
π
, Cov(ε k
,
ε k
)

1
N
(
 
1

2
) c k
21

22

23

 Introduction
 Camera Model
 Particle Filtering Estimation
 Complete System of Video Stabilization
 Simulation and Results
 Conclusion
24

Complete system of video stabilization
• At time k
Frame k
SIFT algorithm x k
PFME
(Particle filteringbased motion estimation) xˆ k

{ s
ˆ k
, k
, T
ˆ k
}
Accumulative motion
Match feature points
Frame k-1
{s
A k
, R
A
, k
T k
A
}
Stailized output
Compensate undesired motion
Kalman filter
25

• SIFT algorithm – Find corresponding pairs
• At time k
It needs three pairs to determine a unique solution
Y X A
A

[X
T
X]

1
X
T
Y
A
 x k

[ s k
, t xk
, t yk
, R
11k
, R
12k
, R
21k
]
T
26

(a) SIFT correspondence from frame 200,201 in outdoor sequence STREET
27

• Important density q(.) is a six-dimensional
Gaussian distribution
• Particles
• In experience , N set to only 30 with better quality than prior distribution set N = 300
28

• N particles have N proposals of transformation matrix ,and N Inverse transform to frame k have N candidate image A i
• Compare these images with k-1 frame A
0
Inverse transform
Point
ˆ at k

1 frame
Point P at k frame match
Point P at k-1frame
29

0
i
• Mean square error
– Difference of gray-scale from pixel to pixel
• Feature likelihood
– Distance of all corresponding feature points
30

Particle filtering for global motion estimation
• Weight for each particle
• Estimation of current state where
31

• At time k-1 to k sˆ k
 sˆ k
,
ˆ k


 tˆ tˆ
• At time 0 to k y x

 , k




11k
21k
12k
22k



Where s is scaling factor , R is rotation matrix and T is translation displacement
32


 u v k

1 k

1


 s
A k

1
R
A k

1

 u v
0
0



T k
A

1

 u v k k


 s
ˆ k k

 u v k 1 k 1


 ˆ k
 s
ˆ k s
A k

1 k
R
A k

1
 s k
A
R
A k

 u v
0
0



T k
A

 u v
0
0


 s
ˆ k k
T k
A

1
 ˆ k
33

Intentional Motion estimation and motion compensation
Implementi ng Kalman filter to get intentiona l rotation matrix R k
D
, translatio n vector T k
D
, and scaling factor s
D k
• Compensate for the unwanted motion
34

Complete system of video stabilization
• At time k
Frame k
SIFT algorithm x k
PFME
(Particle filteringbased motion estimation) xˆ k

{ s
ˆ k
, k
, T
ˆ k
}
Accumulative motion
Match feature points
Frame k-1
{s
A k
, R
A
, k
T k
A
}
Stailized output
Compensate undesired motion
Kalman filter
35

 Introduction
 Camera Model
 Particle Filtering Estimation
 Complete System of Video Stabilization
 Simulation and Results
 Conclusion
36

(a) Original image , (b) Matched-feature-based motion estimation (MFME)
(c) p-norm cost function-based motion estimation (CFME) (d) proposed method 37
(PFME)

(a) Original image , (b) MFME (c) CFME (d) PFME
38

(a) Original image , (b) MFME (c) CFME (d) PFME
39

(a) Original video sequence (ground truth) (b) unstable video sequence (c) PFME
40

T y
?
(a) Motion in horizontal direction (b) Motion in vertical direction
41

Comparison of average MSE and
PSNR for stabilized output
PSNR = peak signal to noise ratio
Large PSNR has low distortion
42

 Introduction
 Camera Model
 Particle Filtering Estimation
 Complete System of Video Stabilization
 Simulation and Results
 Conclusion
43

• We demonstrated experimentally that the proposed particle filtering scheme can be used to obtain an efficient and accurate motion estimation in video sequences.
44

• Constraining rotation matrix projected onto the plane ?(depth change)
• Show using particle filtering can reduce the error variance compared to estimation without particle filtering
• Using both Intensity-based motion estimation method (PFME) and feature-based motion estimation (SIFT) method
45