From Learning Models of Natural Image Patches to
Whole Image Restoration
Daniel Zoran
Interdisciplinary Center for Neural Computation
Hebrew University of Jerusalem
Yair Weiss
School of Computer Science and Engineering
Hebrew University of Jerusalem
Presented by Eric Wang
Duke University
6/18/2012
Introduction
• Patch based learning on images has significant computational
advantages over learning dictionaries over the entire image.
• A primary concern of this paper is the affect the choice of
dictionary priors have on the performance of the model.
• This paper addresses 3 main questions
– (1) Do priors that give high likelihoods yield better patch restoration
performance?
– (2) Do priors that give high likelihoods yield better whole-image
restoration performance?
– (3) Can we learn better priors?
Motivation and Patch Restoration
• Answer to question (1): Priors with higher likelihoods will yield
improved per-patch denoising performance, as shown with
several popular priors
PSNR of restoration on patches vs. dictionary (prior) likelihoods. Trained on
50,000 8x8 patches of natural images of faces. Tested on unseen patches.
From Patches to Whole Image Restoration
• Good patch-based image restoration does not guarantee high
quality image restoration, many reconstruction methods can
generate significant artifacts.
Expected Patch Log-Likelihood
• Choosing random patches can provide a solution to minimize
artifacting, but has the issue that most random patches will
have low likelihood to a given dictionary.
• This paper presents an optimization algorithm that maximizes
expected patch log-likelihood (EPLL) with the constraint that
the reconstructed image be close to the original image.
• The EPLL under prior p is defined as
• Where
image x.
is a mask that extracts the ith random patch from
Cost Function
• Let
be the corruption model on the image where x
is a vectorized image, A defines the corruption model and y is
the vectorized noisy observation.
• The cost function to minimize is then
• Direct optimization of this cost function is intractable, so an
alternative method called half quadratic splitting is used,
where
are a set of per-patch auxiliary variables.
The Corruption Model
• The choice of the matrix A is determined by the application.
• For denoising, A is an identity matrix, and
precision
is the noise
• For deblurring, A is a convolution matrix with a known kernel
• For inpainting, A is a diagonal matrix with zeros for the
missing elements.
Optimization
• The EPLL optimization involves two steps: (1) solving for
given
• And (2) solving for
given , which is dependent on the
dictionary and involves solving a MAP estimate of the most
likely dictionary element for a particular patch.
•
is either set by hand or set to
estimated noise standard deviation in
where
.
is the
Image Restoration
• Answer to question (2): It is shown that priors with higher
patch likelihoods yield improved whole image restoration
Building a better dictionary via the GMM
• Gaussian zero mean data can usually be well represented by
the top-m eigenvectors of its covariance matrix.
• This paper proposes clustering the patches (pixels) via a GMM.
Patches sharing a cluster also share a dictionary.
Sample dictionaries for six mixture components
Building a better dictionary via the GMM
• Answer to question (3): The GMM prior outperforms other
priors in both patch and whole-image restoration.
PSNR values, 200 GMM components
Building a better dictionary via the GMM
• The GMM prior is also shown to outperform the ICA prior in
reconstruction with noise level
using EPLL.
Image Denoising
• 68 images from the Berkeley dataset, 8x8 patches,
comaprison is in PSNR
Image Deblurring
• 68 images from the Berkeley dataset with known blur kernels
and 1% white Gaussian noise (comparison is in PSNR)