Image Deblurring with
Optimizations
Qi Shan
Leo Jiaya Jia
Aseem Agarwala
University of Washington
The Chinese University of Hong Kong
Adobe Systems, Inc.
The Problem
2
An Example
Previous Work (1)
Hardware solutions:
[Ben-Ezra and Nayar 2004]
[Levin et al. 2008]
[Raskar et al. 2006]
4
Previous Work (2)
Multi-frame solutions:
[Jia et al. 2004]
[Petschnigg et al. 2004]
[Rav-Acha and Peleg 2005]
[Yuan et al. 2007]
5
Previous Work (3)
Single image solutions:
[Fergus et al. 2006]
[Levin et al. 2007]
[Jia 2007]
6
Most recent work on Single Image Deblurring
Qi Shan, Jiaya Jia, and Aseem Agarwala
High-Quality Motion Deblurring From a Single Image. SIGGRAPH 2008
Lu Yuan, Jian Sun, Long Quan and Heung-Yeung Shum
Progressive Inter-scale and intra-scale Non-blind Image Deconvolution.
SIGGRAPH 2008.
Joshi, N., Szeliski, R. and Kriegman, D.
PSF Estimation using Sharp Edge Prediction, CVPR 2008.
A. Levin, Y. Weiss, F. Durand, W. T. Freeman
Understanding and evaluating blind deconvolution algorithms. CVPR 2009
Sunghyun Cho and Seungyong Lee, Fast Motion Deblurring.
SIGGRAPH ASIA 2009
And many more...
Some take home ideas
1. Using hierarchical approaches to estimate kernel in
different scales
2. Realize the importance of strong edges
3. Bilateral filtering to suppress ringing artifacts
4. RL deconvolution is good, but we've got better chioces
5. Stronger prior does a better job
6. Deblurring by assuming spatially variant kernel is a good
way to go
Today's topic
How to apply natural image statistics, image local
smoothness constraints, and kernel sparsity prior
in a MAP process
Short discussion on
1. the stability of a non-blind deconvolution process
2. noise resistant non-blind deconvolution and
denoising
Image Global Statistics
…
10
Image Global Statistics
…
11
Image Global Statistics
12
Image Local Constraint
I
L
13
Image Local Constraint
I
L
14
Image Local Constraint
I
L
15
Image Local Constraint
p2
I
L
pL

N
(
d
L
d
I
0
,

)
2
i()
i
i|
1
16
Kernel Statistics
exponentially distributed
fj
pj(f)e
17
Combining All constraints
L
f
n
m
i
n
E
(
L
,
f
)

m
i
n
l
o
g
[
p
(
n
)
p
(
d
L
)
p
(
L
)
p
(
f
)
]
1
2
Two-step iterative optimization
• Optimize L
• Optimize f
18
Optimization Process
Optimize L
Idea: separate convolution

replace dLi with  i

2
E
'
(
L
)

\
S
u
m
(
|
|L
*
f
I
|
|)
2
log p(n)

o
gp
d
L
)||1
1||l
1(
log p1(dL)
2


(
\
S
u
m
(
m
|
|
d
L

d
I
|
|)
)
2
i
i
i 2
log p2 (L)
19
Optimization Process
Optimize L
Idea: separate convolution

replace dLi with  i

2
E
'
(
L
)

\
S
u
m
(
|
|L
*
f
I
|
|)
2
log p(n)
2


(
\
S
u
m
(
m
|
|


d
I
|
|)

o
gp
(
)||1 2
i
i
i 2)
1||l
1
log p1(dL)
log p2 (L)
20
Updating L
Adding a new constraint to make  ~ dL
Removing terms that are not relevant to L


E '( L)  \ Sum(|| L * f  I ||22 ) 
2
2


(
||


d
L
||


(
\
S
u
m
(
|
|


d
I
|
|)
)

|
|l
o
g
p
(

)|
|
2)
2
i
i 2
1
1
1


2
2


(
||


d
L
||
L

a
r
g
m
i
n
\
S
u
m
(
|
|
d
L
*
fd

I
|
|
)
o
p
t
L
2
2)
An easy quadratic optimization problem with a
closed form solution in the frequency domain
21
Updating 
Removing terms that are not relevant to 


E '( )  \ Sum(|| L * f  I ||22 ) 
2
2


(
||


d
L
||


(
\
S
u
m
(
|
|


d
I
|
|)
)

|
|l
o
g
p
(

)
|
|
2)
i
i 2
1
1
1 2


2


a
r
g
m
i
n
|
|
l
o
g
p
(

)
|
|

(
\(
S
u
m
|
|


d
I
|
|
)
)
o
p
t

1
1 1
2
i
i
2

(||dL||22)
22
 
2


a
r
g
m
i
n
|
|
l
o
g
p
(

)
|
|

(
\(
S
u
m
|
|


d
I
|
|
)
)
o
p
t

1 11
2
i i
2

(||dL||22)

a
r
g
m
i
n
(
\
S
u
m
(
E
'
)
)

i
each E ' only contains a single variable Ψi
i
It is then a set of easy single variable
optimization problems
23
Iteration 0
(initialization)
24
Time: about 30 seconds for an 800x600 image
Iteration 8
(converge)
25
A comparison
RL deconvolution
26
A comparison
Our deconvolution
27
Two-step iterative optimization
• Optimize L
• Optimize f
m
i
n
E
(
L
,
f
)

m
i
n
l
o
g
[
p
(
n
)
p
(
d
L
)
p
(
L
)
p
(
f
)
]
1
2
2
E
()|
f
|L
*
f
I
|
|


|
|f|
|
2
1
Optimization with a total variation regularization
28
Results
29
Results
30
31
32
More results
33
More results
34
Today's topic
How to apply natural image statistics, image local
smoothness constraints, and kernel sparsity prior
in a MAP process
Short discussion on
1. the stability of a non-blind deconvolution process
2. noise resistant non-blind deconvolution and
denoising
Stability
Considering the simplest case: Wiener Filtering
T
F
X T
B How about if
FF
I
B B* n



T
*
F
X
*
X

X

n
T
T
F
F

I F
F

I
T
F
And X*  T B*
FF
Stability
Thus

P2
|
|X

X
|
||
|
|
|

C
2
P
P

*2
2
2

where P is the frequency domain representation of F
 2 is the variance of the noise
Observation: the noise in the blur image is magnified in
the deconvolved image. And the Noise Magnification
Factor (NMF) is solely determined by the filter F
Some examples
Some examples
Dense kernels are less stable for deconvolution
than sparse ones
Noise resistant deconvolution and denoising
With Jiaya Jia, Singbing Kang and Zenlu Qin
In CVPR 2010
See you in San Francisco!
Blind and non-blind image deconvolution software
is available online and will be updated soon!
40