Patch-based Image Deconvolution
via Joint Modeling of Sparse Priors
Chao Jia and Brian L. Evans
The University of Texas at Austin
12 Sep 2011
1
Non-blind Image Deconvolution
Reconstruct natural image from blurred version


Camera shake; astronomy; biomedical image reconstruction

2D convolution matrix H and Gaussian additive noise vector n

Maximum a-posteriori (MAP) estimation for vector X


2
Prior model for p(X) for natural images? [Elad 2007]
Optimization method?
Analysis-based modeling [Krishnan 2009]
Prior based on hyper-Laplacian distribution of the spatial
derivative of natural images



3
Linear filtering to compute spatial derivative
Fit (0.5-0.8) and (normalization factor) to empirical data
Patch-based modeling
Sparse coding of patches




Spatial receptive fields of visual cortex [Olshausen 1997]
For 10 10 patches
Learn an overcomplete dictionary
from natural images.
Application in image restoration


Denoising, superresolution
[Yang 2010]


Localized algorithm: patches can
overlap
Use this model in deconvolution?
[Lee 2007]
4
Prior model in natural images
From local to global

5

Slow convergence (EM Algorithm)

Patches should not overlap (Why?)
boundary artifacts
Joint modeling
Take advantage of patch-based sparse representation
while resolving the problems in?
Combine analysis-based prior and synthesis-based prior


Accelerate convergence
Keep consistency on the
boundary of adjacent patches
Patch-based
sparse coding
Sparse spatial
gradient
Keep details and textures
6
Joint modeling

Discard the generative model

Prior probability
sparsity of
gradients

7
sparsity of
representation
coefficients
After training, we fix the parameters for all images
compatibility
term
MAP estimation using the joint model
Problem:

likelihood
prior
Iteratively updating w and X until convergence



8
w sub-problem
small-scale L1 regularized square loss minimization
X sub-problem
Half-quadratic splitting [Krishnan 2009]
Experimental results
Initialization: Wiener estimates / blurred images
Dictionary: learned from Berkeley Segmentation database



Patch size 12 12
Prior parameters:
Runtime: (Matlab) 16s with Intel Core2 Duo CPU @2.26GHz
Experiment settings:



9
Experimental results
2
4
6
8
ISNR
comparison
test 1
test 2
[Krishnan 2009]
test 3
[Portilla 2009]
test 4
proposed
0.8
0.85
0.9
test 1
SSIM
comparison
test 2
[Krishnan 2009]
test 3
[Portilla 2009]
test 4
proposed
10
PASCAL Visual
Object Classes
Challenge (VOC)
2007 database
Experimental results
11
Experimental results
[Krishnan 2009]
Original image
[Portilla 2009]
keeps more
brick textures
Blurred image
12
Proposed
Experimental results

Textures zoomed in
Original image
[Krishnan 2009]
13
[Portilla 2009]
Proposed
Conclusions

Global model for MAP estimation


Joint model of image pixels and representation
coefficients



Sparsity of spatial derivative (analysis-based)
Sparsity of representation of patches in overcomplete
dictionary (synthesis-based)
Iterative algorithm


Able to solve general non-blind image deconvolution
converges in a few iterations
Matlab code for the proposed method is available at
http://users.ece.utexas.edu/~bevans/papers/2011/sparsity/
14
References





[Elad 2007] M. Elad, P. Milanfar and R. Rubinstein, “Analysis versus
synthesis in signal priors”, Inverse Problems, vol. 23, 2007.
[Krishnan 2009] D. Krishnan and R. Fergus, “Fast image deconvolution using
hyper-Laplacian priors,” Advances in Neural Information Processing
Systems, vol. 22, pp. 1-9, 2009.
[Olshausen 1997] B.A. Olshausen and D.J. Field, “Sparse coding with an
overcomplete basis set: a strategy employed by V1,” Vision Research, vol.
37, no. 23, pp. 3311-3325, 1997.
[Portilla 2009] J. Portilla, “Image restoration through L0 analysis-based
sparse optimization in tight frames,” in Proc. IEEE Int. Conf. on Image
Processing, 2009, pp. 3909-3912.
[Yang 2010] J. yang, J. Wright, T.S. Huang and Y. Ma, “Image superresolution via sparse representation,” IEEE Trans. on Image Processing, vol.
19, no. 11, pp. 2861-2873, 2010.
15
 Thank you!
16
w sub-problem
patches do not overlap
small-scale l1
regularized square loss
minimization
17
X sub-problem


Conjugate gradient
iteratively reweighted least squares
Half-quadratic splitting [Krishnan 2009]
auxiliary
variable
component-wise
quartic function
18
No need to
solve the
equation
MAP estimation using the joint model
blurred image;
noise level; blurring
kernel; initialization
of recovered image
finish
X sub-problem
Update the coefficient
of patches
(w sub-problem)
X converges?
Set α=α0
α>αmax ?
Update auxiliary
variable Y
(quartic equation)
19
α=kα
Update image X
(FFT)
Image Quality Assessment

Full reference metric
ISNR -- increment in PSNR (peak signal-to-noise ratio)

SSIM -- structural similarity [Wang 2004]

20
Prior model of natural images

Analysis-based prior



Fast convergence
Over smooth the images
Synthesis-based prior (patch-based sparse
representation)




21
Dictionary well adapted to nature images
Captures textures well
Slow convergence
Boundary artifacts
Computational complexity

Computational complexity



For each iteration:
N is the total number of pixels in the image
Average runtime comparison
22
[Krishnan 2009]
[Portilla 2009]
Proposed
2s
15s
16s