Unnatural L0 Representation
for Natural Image Deblurring
Speaker: Wei-Sheng Lai
Date: 2013/04/26
Outline
1.
2.
3.
4.
Introduction
Related work
L0 Deblurring
Conclusion
2
1. Introduction
• Form of image blur :
1. Object motion
2. Camera Shake
3. Out of focus (defocus)
•
Blur model:
𝐵 =𝐿⊗𝐾+𝑁
B: blurred(observed) image
L: latent(sharp) image
K: blur kernel
N: noise
⨂: convolution
Point Spread Function (PSF)
3
1. Introduction
• Ill-posed problem:
observation (B) < unknown variables (L + K)
4
1. Introduction
• Early method:
1. Richardson–Lucy deconvolution (RL) [1][2]
𝐿𝑡+1
=
𝐿𝑡
𝐵
.∗ 𝐾 ⊗ 𝑡
𝐿 ⊗𝐾
𝐾: flipped blur
kernel
2. Wiener filter [3]
𝐿(𝐹) = 𝐵(𝐹) .∗

𝐾 ∗ (𝐹)
2
𝐾 𝐹 2+𝜎 𝐴
𝜎 : noise ratio
A : constant
Both are known to be sensitive to noise.
[1] Richardson, William Hadley. "Bayesian-based iterative method of image restoration." JOSA 62.1 (1972): 55-59.
[2] Lucy, L. B. "An iterative technique for the rectification of observed distributions."The astronomical journal 79 (1974): 745.
[3] Wiener, Norbert. Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications. Technology
5
Press of the Massachusetts Institute of Technology, 1950.
1. Introduction
• Recent framework: Maximum-a-Posteriori (MAP)
𝐿∗ , 𝐾 ∗ = 𝑎𝑟𝑔 min 𝐵 − 𝐿 ⊗ 𝐾
𝐿,𝐾
2
2 + ρ𝐿
𝐿 + ρ𝐾 𝐾
– ρ𝐿 𝐿 : prior of latent image
– ρ𝐾 𝐾 : prior of kernel
• Non-linear problem, iterative optimization :
𝐿∗ = 𝑎𝑟𝑔 min 𝐵 − 𝐿 ⊗ 𝐾
𝐿
𝐾 ∗ = 𝑎𝑟𝑔 min 𝐵 − 𝐿 ⊗ 𝐾
𝐾
2
2 + ρ𝐿
2
2 + ρ𝐾
𝐿
𝐾
6
2. Related work
• Fergus et al. Siggraph 2006 [4]
– Heavy tails distribution of nature image gradient
– Assume kernel prior as Gamma distribution
𝑥 𝑎 𝑒 −𝑥/𝑏
𝑃 𝑥 𝑎, 𝑏 =
𝑎! 𝑏 𝑎+1
[4] R. Fergus et al, “Removing camera shake from a single photograph,” Siggraph 2006
7
2. Related work
• Prior (regularization) :
– Gaussian prior (L2 regularization) [5]:
𝐸(𝐿) = 𝐵 − 𝐿 ⊗ 𝐾 22 + 𝜆 𝛻𝐿
– TV-L1 prior [6]:
𝐸(𝐿) = 𝐵 − 𝐿 ⊗ 𝐾
– Sparse prior [7]:
𝐸(𝐿) = 𝐵 − 𝐿 ⊗ 𝐾
2
2
+ 𝜆 𝛻𝐿
2
2
+ 𝜆 𝛻𝐿
2
2
1
𝛼
,𝛼
[5] S Cho et al, “Fast motion deblur,” Siggraph 2009
[6] Xu, Li, and Jiaya Jia. "Two-phase kernel estimation for robust motion deblurring." ECCV 2010.
[7] Levin, Anat, et al. "Image and depth from a conventional camera with a coded aperture." ACM TOG 2007
≤1
8
2. Related work
• Q.Suan et al. Siggraph 2008 [8]
– N and 𝜕𝑁 should follow the zero-mean Gaussian
distribution
𝜕∗𝐵 − 𝜕∗𝐿 ⊗ 𝐾
E L =
2
2
+ 𝜆1 𝜌 𝜕𝑥 𝐿 + 𝜌 𝜕𝑦 𝐿
𝜕∗
+ 𝜆2
𝜕𝑥 𝐿 − 𝜕𝑥 𝐵
2
2
+ 𝜕𝑦 𝐿 − 𝜕𝑦 𝐵
2
2
+ 𝐾
[8] Q. Shan et al, “High quality motion deblurring from a single image,” Siggraph 2008
1
1
9
2. Related work
• Cho et al. Siggraph 2009 [5]
– Accelerate the deblurring procedure by first estimating a
predicted image and using L2 regularization
• Kernel estimation :
𝜕∗𝐵 − 𝜕∗𝑃 ⊗ 𝐾
𝐸 𝐾 =
2
2
+𝛽 𝐾
2
2
+ 𝜆 𝛻𝐿
2
2
𝜕∗
• Image deconvolution:
𝜕∗𝐵 − 𝜕∗𝐿 ⊗ 𝐾
𝐸 𝐿 =
2
2
𝜕∗
[5] S Cho et al, “Fast motion deblur,” Siggraph 2009
10
2. Related work
• Anat Levin et al. CVPR 2009 [9] :
– MAP x,k approach will favor blur image with delta kernel.
𝐿∗ , 𝐾 ∗ = 𝑎𝑟𝑔 min 𝐵 − 𝐿 ⊗ 𝐾
𝐿,𝐾
2
2 +𝜆
𝛻𝐿
2
2
– Estimate kernel K first, then use non-blind deconvolution to
solve the latent image.
[9] Levin, Anat, et al. "Understanding and evaluating blind deconvolution algorithms." CVPR 2009.
11
Unnatural L0 Sparse Representation
for Natural Image Deblurring
12
3. L0 Deblurring
• Li Xu et al. CVPR 2013 [10]
– Predict image with L0 optimization
• L0-norm:
𝜌 𝑥 = 𝑥
0
=
0,
1,
𝑥 =0
𝑥 ≠0
• Approximate L0 sparsity function:
1
2
𝛻
𝐼
,
∗
𝜙 𝛻∗ 𝐼 = 𝜖 2
1,
𝑖𝑓 𝛻∗ 𝐼 ≤ 𝜖
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
[10] Xu, Li, Shicheng Zheng, and Jiaya Jia. "Unnatural L0 Sparse Representation for Natural Image Deblurring.” CVPR 2013
13
3. L0 Deblurring
• Main objective function:
𝑚𝑖𝑛
𝐾⊗𝐼−𝐵
2
2
+𝜆
𝜙0 𝜕∗ 𝐼 + 𝛾 𝐾
2
2
∗∈{𝑥,𝑦}
where 𝜙0 𝜕∗ 𝐼 =
𝑖 𝜙(𝜕∗ 𝐼𝑖 ),
1
𝜖2
𝜙 𝛻∗ 𝐼 =
𝛻∗ 𝐼 2 , 𝑖𝑓 𝛻∗ 𝐼 ≤ 𝜖
1,
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
• Iteratively solve:
𝐼 (𝑡+1)
= 𝑎𝑟𝑔 min
𝐼
𝐾 (𝑡+1) = 𝑎𝑟𝑔 min
𝐾
𝐾 (𝑡)
⊗𝐼−𝐵
2
2
+𝜆
𝐾 ⊗ 𝐼 (𝑡+1) − 𝐵
𝜙0 𝜕∗ 𝐼
∗∈{𝑥,𝑦}
2
+𝛾 𝐾
2
2
2
[10] Xu, Li, Shicheng Zheng, and Jiaya Jia. "Unnatural L0 Sparse Representation for Natural Image Deblurring.” CVPR 2013
14
3. L0 Deblurring
• Solving
𝐼 (𝑡+1) = 𝑎𝑟𝑔 min
𝐼
where 𝜙0 𝜕∗ 𝐼 =
𝐾 (𝑡) ⊗ 𝐼 − 𝐵
𝑖 𝜙(𝜕𝑥 𝐼𝑖 )
,
2
2
+𝜆
𝜙 𝜕∗ 𝐼 =
𝜙0 𝜕∗ 𝐼
∗∈{𝑥,𝑦}
1
𝜖2
𝜕∗ 𝐼 2 , 𝑖𝑓 𝜕∗ 𝐼 ≤ 𝜖
1,
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
• Equivalent to solving
𝐼 (𝑡+1) = 𝑎𝑟𝑔 min
𝐼,𝑤
𝐾 (𝑡) ⊗ 𝐼 − 𝐵
2
+𝜆
2
0,
𝑤∗𝑖 =
𝜕∗ 𝐼,
𝑤∗𝑖
∗∈{𝑥,𝑦} 𝑖
𝑖𝑓 𝜕∗ 𝐼 ≤ 𝜖
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
0
+
1
(𝑡) − 𝑤
𝜕
𝐼
∗
∗𝑖
𝜖2
2
𝜖 ∈ {1, 2−1 , 4−1 , 8−1 }
[10] Xu, Li, Shicheng Zheng, and Jiaya Jia. "Unnatural L0 Sparse Representation for Natural Image Deblurring.” CVPR 2013
15
3. L0 Deblurring
𝐹 𝑥 =
𝐹 𝐵𝑀 .∗ 𝐹 𝑌 + 𝜆/𝜖 2 𝐹 𝜕𝑥 .∗ 𝐹 𝑤𝑥 + 𝐹 𝜕𝑦 .∗ 𝐹 𝑤𝑦
𝐹 𝐵𝑀 .∗ 𝐹 𝐵𝑀 + 𝜆/𝜖 2 𝐹𝐷2
𝐾
(𝑡+1)
= 𝑎𝑟𝑔 min
𝐾
𝐾⊗𝐼
(𝑡+1)
(𝑡+1)
𝑘 (𝑡+1)
=
𝐹 −1
𝐹(𝐴𝑀
𝐹
𝑘 (𝑡+1) = 𝑘 (𝑡)
−𝐵
2
+𝛾 𝐾
2
2
2
) .∗ 𝐹(𝑦)
(𝑡+1)
𝐴𝑀
2
+𝛾
𝛼
𝐴𝑇𝑅 𝑦
(𝐴𝑇𝑅 𝐴𝑅 + 𝛾)𝑘
𝑛
[10] Xu, Li, Shicheng Zheng, and Jiaya Jia. "Unnatural L0 Sparse Representation for Natural Image Deblurring.” CVPR 2013
16
3. L0 Deblurring
Unnatural
Fast Hyper-Laplacian
deconvolution (𝐿0.5 norm) [11]
Representation
Input image
Deblurring
result
L0 optimization
Predict map
kernel
[11] Krishnan, Dilip, and Rob Fergus. "Fast image deconvolution using hyper-Laplacian priors." ANIPS 2009
17
3. L0 Deblurring
• Other results
18
3. L0 Deblurring
• Advantage of L0 deblurring:
– Fast convergence
– High quality
19
4. Conclusion
• A naïve MAP x,k estimation will fail.
𝐿∗ , 𝐾 ∗ = 𝑎𝑟𝑔 min 𝐵 − 𝐿 ⊗ 𝐾
𝐿,𝐾
2
2 +𝜆
𝛻𝐿
2
2
• How to estimate correct kernel is important.
• It is not as simple as what I have shown, there are
many implementation details.
20
Thanks for Attention !
21