Removing blur due to camera shake
from images.
William T. Freeman
Joint work with Rob Fergus, Anat Levin, Yair Weiss, Fredo Durand,
Aaron Hertzman, Sam Roweis, Barun Singh
Massachusetts Institute of Technology
Overview
Original
Our algorithm
Close-up
Original
Naïve Sharpening
Our algorithm
Let’s take a photo
Blurry result
Slow-motion replay
Slow-motion replay
Motion of camera
Image formation process

=
Blurry image
Input to algorithm
Model is approximation
Blur
kernel
Sharp image
Desired output
Convolution
operator
Why is this hard?
Simple analogy:
11 is the product of two numbers.
What are they?
No unique solution:
11 = 1 x 11
11 = 2 x 5.5
11 = 3 x 3.667
etc…..
Need more information !!!!
Multiple possible solutions
Sharp image
Blur kernel
=

=

=

Blurry image
Is each of the images that follow
sharp or blurred?
Another blurry one
Natural image statistics
Characteristic distribution with heavy tails
Histogram of image gradients
Blury images have different statistics
Histogram of image gradients
Parametric distribution
Histogram of image gradients
Use parametric model of sharp image statistics
Uses of natural image statistics
•
•
•
•
•
•
Denoising [Roth and Black 2005]
Superresolution [Tappen et al. 2005]
Intrinsic images [Weiss 2001]
Inpainting [Levin et al. 2003]
Reflections [Levin and Weiss 2004]
Video matting [Apostoloff & Fitzgibbon 2005]
Corruption process assumed known
Existing work on image deblurring
Software algorithms:
– Extensive literature in signal processing community
– Mainly Fourier and/or Wavelet based
– Strong assumptions about blur
 not true for camera shake
Assumed forms of blur kernels
– Image constraints are frequency-domain power-laws
A focus on image
constraints, not image
priors
Some image constraints/priors
Toy example:
1.0
0.0
observed “image”:
Toy example:
1.0
0.0
observed “image”:
Toy example:
observed “image”:
1.0
0.0
Three sources of information
1. Reconstruction constraint:

Estimated sharp image
=
Estimated
blur kernel
2. Image prior:
Distribution
of gradients
Input blurry image
3. Blur prior:
Positive
&
Sparse
Prior on image gradients
(mixture of Gaussians giving a
Laplacian-like distribution)
Distribution of gradients (log-scale)
Green curve is our mixture of gaussians fit.
Prior on blur kernel pixels
(mixture of exponentials)
P(b)
b
How do we use this information?
Obvious thing to do:
– Combine 3 terms into an objective function
– Run conjugate gradient descent
– This is Maximum a-Posteriori (MAP)
Maximum A-Posteriori
y – observed blurry image
x – unobserved sharp image
b – blur kernel
Likelihood
Latent image prior
Blur prior
Sparse and
i – image patch index
f – derivative filter
Assumption: all pixels independent of one another
Results from MAP estimation
Input blurry
image
Maximum a-Posteriori (MAP) Our method: Variational Bayes
Variational Bayes
http://citeseer.ist.psu.edu/cache/papers/cs/16537/http:zSzzSzwol.ra.phy.cam.ac.ukzSzjwm1003zSzspringer_chapter8.pdf/miskin00ensemble.pdf
Miskin and Mackay, 2000
Setup of variational approach
Need likelihood and prior in same space, so use gradients:
Likelihood
Prior on latent image gradients – mixture of Gaussians
Prior on blur elements – mixture of Exponentials
i – image pixel
j – blur pixel
We use C=4, D=4
Also have Gamma hyperpriors on
Variational inference
• Approximate posterior
with
• Cost function
• Assume
is Gaussian on each pixel
is rectified Gaussian on each pixel
• Use gradient descent, alternating between updating
marginalizing out over
and vice versa
• Adapted code from Miskin & Mackay 2000
while
Variational Bayesian method
Based on work of Miskin & Mackay 2000
Keeps track of uncertainty in estimates of image and blur by
using a distribution instead of a single estimate
Helps avoid local maxima and over-fitting
Variational Bayesian method
Objective function for a single variable
Score
Maximum
a-Posteriori (MAP)
Variational
Bayes
Pixel intensity
MAP vs Variational
MAP
Variational
MAP using variational initialization
Blurry synthetic image
Inference – initial scale
Inference – scale 2
Inference – scale 3
Inference – scale 4
Inference – scale 5
Inference – scale 6
Inference – final scale
Our output
Ground truth
Matlab’s deconvblind
True kernel
Estimated kernel
Tried the same algorithm on an image of
real camera blur, with very similar blur
kernel
Failure!
Does camera shake give a stationary blur kernel?
8 different people, handholding camera, using 1 second exposure
Whiteboard scene:
View of the dots at each corner in photos taken by 4 people
Person 2
Person 1
Top
left
Top
right
Bot.
left
Bot.
right
Person 3
Person 4
Tonescale: output from camera
Linear response to light intensities looks like this
Overview of algorithm
Input image
1. Pre-processing
2. Kernel estimation
-
Multi-scale approach
3. Image reconstruction
-
Standard non-blind deconvolution routine
Preprocessing
Input image
Convert to
grayscale
Remove gamma
correction
User selects patch
from image
Bayesian inference
too slow to run on
whole image
Infer kernel
from this patch
Initialization
Input image
Convert to
grayscale
Remove gamma
correction
User selects patch
from image
Initialize 3x3
blur kernel
Blurry patch
Initial image estimate
Initial blur kernel