Robust Alignment by Sparse and
Low-rank Decomposition for
Linearly Correlated Images
Zhihui Zhu
Instructor: Prof. Hoff
Colorado School of Mines
December 2, 2013
1
Outlier
• Introduction
• Previous work on alignment
• Alignment by sparse and low rank
decomposition
• Experiment results
• Conclusion
2
Introduction
• Multiple images of an object
• Challenging existing vision
alg.: face detection etc.
• Because of:
illumination variation, partial
occlusion, no alignment
3
Recognition Pipeline
Detection
Alignment
Identification
Recognition Pipeline[1]:
Hw 2
Alignment: using affine/geometric
transformation to match
4
Batch Image Alignment
• Given many images of an object, simultaneously
align them to a fixed canonical template
• Mathematically:
Given
Recover
observations {I i }
images {I i0 }
=
I i ( I i0 + ei )  τ i−1
transformaitons {τ }
−1
i
5
Outlier
• Introduction
• Previous work on alignment
• Alignment by sparse and low rank
decomposition
• Experiment results
• Conclusion
6
Previous work
• Learned-Miller’s congealing alg.[2]:
minimize the sum of entropies
• Least squares congealing alg.[3]:
Minimize the sum of squared distances
Rank 1
Low rank
• Minimize log-determinant measure [4]
7
Outlier
• Introduction
• Previous work on alignment
• Alignment by sparse and low rank
decomposition
• Experiment results
• Conclusion
8
Modeling Misalignments as Domain
Transformations
• I1 and I 2 misaligned images, exists τ ∈ Aff(2)
=
I 2 ( x , y ) (=
I1  τ )( x, y ) I1 (τ ( x, y ))
• 2-D Affine group (Aff(2)):
G
0

t
1
Hw 2
 x   g11 g12 t x  u 
 y  =  g g t  v 
   21 22 y   
1  0 0 1  1 
 1.23 0.49 -55.14 

τ =  −0.11 0.87 23.19 
 0
0
1 
9
Matrix Rank as Measure of Image
Similarity
• Well-aligned images are linearly correlated
0
0
m×n
=
A
vec
I
⋅⋅⋅
vec
I
∈
R
[
(
)
|
|
(
)]
• Matrix
1
n
should be row rank
 If {I i0 } are the same
rank one
0
{
I
 If i } are images of convex Lambertian object
under varying illumination
rank 9 approx. [5]
Fig. The same face, under two different lighting conditions[5].
10
Modeling Corruption as Large,
Sparse Errors
• Corruptions or occlusions break low-rank
structure
• Fortunately, affect only a small fraction
model as sparse errors
11
Robust Alignment of Correlated
Images
• Given
• Recover
observations {I i }
images {I i0 }
=
I i ( I i0 + ei )  τ i−1
transformaitons {τ }
−1
i
min rank ( A) + γ || E ||0 s.t. D  τ =
A+ E
A, E ,τ
=
D  τ [vec( I1  τ 1 ) | ⋅⋅⋅ | vec( I n  τ n )]
12
Convex Relaxation
NP-hard
• Rank(A)
• l0 -norm
Relaxation method
• Nuclear norm || A ||∗ = ∑ σ i ( A)
• l1 -norm || A ||1
i
13
Iterative Linearization
• Approximate by Taylor Series
∂
=
;J
f (x + ∆
x) f ( x) + J ⋅ (∆x) + h.o.t=
f (ξ ) |ξ = x
∂ξ
• Here
∂
I  (τ + ∆τ ) ≈ I τ + J ∆τ ; J = ( I  ξ ) |ξ =τ
∂ξ
• And
D  (τ + ∆τ ) ≈ D τ + ∑ J i ∆τ i eiT
i
∂
T
=
Ji =
( Ii  ξ ) |ξ =τ i , ei [0 0 1 0 0]
∂ξ
14
Iterative Convex Programming
• Leads to a convex optimization problem:
T
A+ E
min || A ||∗ +γ || E ||1 s.t. D  τ + ∑ J i ∆τ i ei =
A, E , ∆τ
i
Main Iteration
Initialization
k =0
τ 1 , ,τ n
k= k + 1
1. Compute Jocobian : J i
2. Warp and Normalize :
D τ = [
vec ( I 1  τ 1 )
|| vec ( I 1  τ 1 ) ||2
||
vec ( I n  τ n )
|| vec ( I n  τ n ) ||2
]
3. Solve above linearized convex optimization :
Accelerated Proximal Gradient (APG)
4. Update Transforamtion : τ = τ + ∆τ
4.
No
∆τ 2 ≤ ε
Yes
Stop
15
Outlier
• Introduction
• Previous work on alignment
• Alignment by sparse and low rank
decomposition
• Experiment results
• Conclusion
16
Alignment Example
Original images
Aligned images
17
Alignment Example
Low rank part A
average of unaligned D
Sparse errors E
average of aligned D
average of A
18
Comparison
 
Rotation [ −10 ,10 ]
x- and y-translations: [ −3, 3] pixels
Occluded by 12 x12 batch
(a) Original perturbed and corrupted images
(b) Alignment results by [4]
(c) Alignment results by RASL
19
Comparison
(d) Alignment results by RASL: upper: low rank part; down: sparse errors
Mean error
Error std.
Max error
Initial misalignment
2.5
1.03
4.87
[23]
1.66
0.85
4.02
RASL
0.48
0.23
1.07
20
Removing occlusions
Original homograph images
Aligned images
21
Removing occlusions
Reconstructed images A
average of unaligned D
Removed occlusions E
average of aligned D
average of A
22
Outlier
• Introduction
• Previous work on alignment
• Alignment by sparse and low rank
decomposition
• Experiment results
• Conclusion
23
Conclusion
• Seeks a set of image domain transformations
• Low rank + sparse structure seems a good
model for aligned images
• Experiments verify this point
• Extension to large scale situations.
• Online robust image alignment?
24
25
Reference
[1] Gary B. Huang, Vidit Jain, and Erik Learned-Miller. Unsupervised joint
alignment of complex images. ICCV, 2007.
[2] E. Learned-Miller. Data driven image models through continuous joint
alignment. PAMI, 2006.
[3] M. Cox, S. Lucey, S. Sridharan, and J. Cohn. Least squares congealing for
unsupervised alignment of images. CVPR, 2008.
[4] A. Vedaldi, G. Guidi, and S. Soatto. Joint alignment up to (lossy)
transforamtions. CVPR, 2008.
[5] R. Basri and D. Jacobs. Lambertian reflectance and linear subspaces.
PAMI, 2003.
[6] http://perception.csl.illinois.edu/matrix-rank/rasl.html
26