Leica - CNIC 1st Practical School in Super-Resolution Microscopy, Centro Nacional de Investigaciones Cardiovasculares Carlos III (CNIC)
Image Restoration by Deconvolution:
Concepts and Applications
Chong Zhang
SIMBioSys, Depertment of Information and Communication Technologies
Universitat Pompeu Fabra
15th March, 2016
What is deconvolution in your mind?
Convolution
Deconvolution
Point Spread Function
Noise
Fourier Transform
Spatial resolution
Pixel size
Rayleigh Criterion
Airy disk
Numerial Aperture
Refractive Index
Wavelength
How do you perform deconvolution?
Fiji plugins
Huygens Professional
MC AutoquantX
…
What is deconvolution (in microscopy)?
Deconvolution is a computational technique allowing to partly compensate for the
image distortion caused by a microscope.
The betterment can be significant both in terms of attenuation of the out of focus light
and increase of the spatial resolution.
It was first devised at the MIT for seismology (Robinson, Wiener, early 50’), then applied
to astronomy and finally found its way to 3D optical fluorescence microscopy (Agard
1984).
It should not be seen as a “black box” to enhance image quality since it can introduce
artifacts or further degrade low quality images.
It is compatible with quantitative measurements (should even improve).
It works best for thin (<50 um), optically transparent, fixed, bright samples.
Challenging for live microscopy: short exposure (limit motion blur), objective adapted
to medium (limit spherical aberrations).
Courtesy of S. Tosi
Ideal case
Image
Detector
Ideal case Objective
Specimen
Courtesy of P. Bankhead
Reality
Image
- Blurred
Detector
More realis-c Objective
Specimen
Courtesy of P. Bankhead
Reality
Image
- Blurred
- Noisy
Detector
More realis-c Objective
Specimen
Courtesy of P. Bankhead
Noises along the optical train in digital microscopy
Courtesy of S. Tosi
Convolution
Convolution consists of replacing each point in the original object with its blurred image in
all dimensions and summing together overlapping contributions from adjacent points to
generate the resulting image
Filter kernel
!"
Original
Filtered image
image
Originalimage
image
Filtered
“stamping” the kernel on each pixel of the image
the kernel is scaled (multiplied) by the intensity of the central pixel
Accumulated (summed) in the output image.
Courtesy of S. Tosi
Convolution
Convolution consists of replacing each point in the original object with its blurred image in
all dimensions and summing together overlapping contributions from adjacent points to
generate the resulting image
=
=
A 3D kernel is a stack holding
the filter coefficients
Courtesy of S. Tosi
Fourier Transform (FT) vs convolution
If we take the FT of the
equation, the is replaced by
multiplication,
thus image restoration might
be achievable by:
dividing the FT of the image
by the FT of the kernel and
then taking the inverse
Fourier transform.
=
Spatial
Domain
Fourier
Domain
X
=
Courtesy of S. Tosi
Point Spread Function (PSF)
A record of how much the image created by a microscope spreads/blurs an object of a single
point (thus determines the way in which images of objects blur into each other in the final
image).
Widefield PSF
Bankhead 2014
Cannell 2006
Theoretical PSF vs Measured PSF
Measured PSF: an image resulting from a single small spherical fluorescent bead (smaller than
the optical resolution, thus forms effectively a point source of light)
Theoretical PSF: generated by a computer program, after input of values describing the optical
system – the magnification, NA of the objective, the illuminating and emitted wavelengths, and
the refractive index of the objective lens immersion medium. It gives an indication of the best
possible resolution for a given objective but these limits are not achievable.
Real PSFs are typically >20% bigger than calculated versions.
Measured PSF
(experimental PSF)
Parton 2006
Theoretical PSF
Bankhead 2014
Courtesy of S. Tosi
Gated 3D STED STED /GS
<50 <130 70 70 560 <130 Courtesy of N. Garin
PSF in the focal plane (where most things are measured)
Airy disk Spatial resolution:
radius of the smallest point source in the
image, i.e. first minimum of Airy disk
Parton 2006
Bankhead 2014
Widefield PSF: 100nm beads, excitation 520nm, emission 617nm
Rayleigh criterion
λ : fluorophore emission wavelength
NA : objective numerical aperture
n : refractive index of the objective lens
immersion medium
NA = n sinθ
NA can never exceed n, which itself has
fixed values (e.g. 1.0 for air, 1.33 for water,
or 1.52 for oil)
Bankhead 2014
Notes:
1. Rayleigh criterion has not taken
into account the effects of:
brightness, pixel size, noise
2. High NAs are possible when the
immersion refractive index is
high
What else does measured PSF tell us?
Asymmetry
radial (x-y):
commonly misalignment of optical
components about the z-axis,
either as tilt or decentration
along the optical axis (z-axis):
commonly due to spherical
aberration, which may result from
refractive index mismatches
between the objective, immersion
medium, and sample or tube
length/coverslip thickness errors.
Pawley 2006
Notes:
1. The immersion refractive index should match the refractive index of the medium surrounding the sample, to
avoid spherical aberration
2. Item 1 is often strongly preferable to using the highest NA objective available, as it is usually better to have a
larger PSF than a highly irregular one.
Deconvolution principle
Noise
Digital filter
Microscope PSF
Deconvolved image
Object (sample)
Courtesy of S. Tosi
=R
=D
The deconvolution filter F should “undo” the effect of the microscope PSF H by processing the
sampled image R, ideally D = S.
Deconvolution principle
Noise
Digital filter
Microscope PSF
Deconvolved image
Object (sample)
Courtesy of S. Tosi
=R
=D
The deconvolution filter F should “undo” the effect of the microscope PSF H by processing the
sampled image R, ideally D = S.
Classification of deconvolution methods
Nearest neighbors
Deblurring
subtractive
No neighbors
Fast
Software
available
Not for
2D only
quantitative
No need PSF intensity
measures
Tikhonov-Miller filter
Linear
inverse
Regularized
inverse
Fast
3D
Software
available
Needs PSF
Wiener filter
Jansson van Cittert
3D
Needs PSF
Constrained
iterative
Richardson-Lucy
Lifting the Fog: Image Restoration by Deconvolution, Parton 2006
3D
Estimates
PSF
Not for
super
resolution
Trade-off
between
sharpness
& noise
Use noise
models
Slow
Good 3D
results
Adaptive blind
Do not
count
for noise
May not
be always
quantitative
Also for
super
resolution
Linear deconvolution: inverse filter deconvolution
For example, assuming H known, F linear (convolution) and no noise (N = 0) leads to:
But in practice…
Noise enhancement ruins our efforts!
2,4·∙107 4,1·∙10-­‐8 1 A very simple model
for the PSF H
(Gaussian std = 1 pixel)
H power spectrum (log display)
overlaid with raw values
1 H-1 power spectrum (log display)
overlaid with raw values
Courtesy of S. Tosi
Inverse filter deconvolution
H is a Gaussian with
std = 2 pixels
+N
H
H-1
Original image S
Noise std = 10-4
S after convolution by H
Noise std = 10-12
No noise
Courtesy of S. Tosi
Second try: regularized inverse
4,1·∙10-­‐8 100 7 2,4·∙10
1 A very simple model
for the PSF H
(Gaussian std = 1 pixel)
H power spectrum (log display)
overlaid with raw values
1 (H-1)reg (1% clipping)
power spectrum (log display)
overlaid with raw values
Courtesy of S. Tosi
Regularized Inverse Filter Deconvolution
+N
H
Original image S
S after convolution by H
(H-1)trunc
Noise std = 10-4
Courtesy of S. Tosi
Third Try: Wiener Filter
The Golden Linear Deconvolution Trade-off
Coming back to:
Minimizing the expectation of ||E|| over all possible noise realizations assuming a white Gaussian noise:
Wiener filter
Bands free of noise: |N(u,v)| = 0 ! F(u,v) = H(u,v)-1
(inverse filter)
Strong noise bands: |N(u,v)|! ∞ ! F(u,v) ! 0
(cut-off)
Intermediate bands:
best trade-off
As noise at certain frequencies increases, the signal-­‐to-­‐noise ra-o drops, so F also drops. This means that the Wiener filter attenuates frequencies dependent on their
signal-to-noise ratio.
Courtesy of S. Tosi
Wiener Deconvolution
Regularized inverse filter result
Noise std = 10-4
Wiener filter result
Noise std = 10-4
Courtesy of S. Tosi
Non-Linear Deconvolution
The best deconvolution algorithms for 3D microscopy are typically non-linear.
Principle of Maximum A Priori algorithms (MAP):
The second equality comes from Bayes theorem.
In the optimization S is usually constrained to be positive and somehow spatially smooth (TV
regularization term) " Pr(S).
The statistical distribution of the noise has to be known to derive the maximum likelihood term Pr(R|S)
" the algorithm is tuned to a particular noise (e.g. Poisson or Gaussian noise).
There is usually no known analytical solution to the problem, the algorithms proceeds by iterations
(candidate Si at iteration i) to refine the estimate of the data at each iteration.
The Richardson-Lucy algorithm is among the most well known MAP deconvolution algorithm.
Some algorithms also simultaneously estimate the PSF from the sampled image (blind deconvolution).
Courtesy of S. Tosi
Quantification after deconvolution
Ideally: relocate signal to the point of origin in 3D, thus conserve the
sum of fluorescence signal. It improves quantification!
In practice: different algorithms have more or less compromises
Quantitative intensity measurements, e.g. intensity ratio: controls, also
report on un-deconvolved data for comparison
Quantitative positional or structural analysis, e.g. centroid, tracking, volume
analysis, (object based) colocalisation, etc: relatively less critical the choice
For all analysis:
Deconvolution process comparable between datasets
Compare with control/un-deconvolved data
Understand algorithm used and choose most suitable
Report possible artifacts and confirm it, if possible
Deconvolution tools
Fiji plugins
Diffraction PSF 3D
PSF generator
Parallel iterative deconvolution (http://imagej.net/Parallel_Iterative_Deconvolution): 4 deconvolution algorithms
Parallel spectral deconvolution (http://imagej.net/Parallel_Spectral_Deconvolution): not iterative, no constraint e.g. nonnegativity
Iterative Deconvolve 3D (http://imagej.net/Iterative_Deconvolve_3D): non-negative, iterative, similar to WPL algorithm.
The execution is way slower on modern (multicore) computers but the memory requirement is less stringent
DeconvolutionLab (http://bigwww.epfl.ch/algorithms/deconvolutionlab/): different algorithms including a custom version
of the thresholded Landweber algorithm
Squassh (http://imagej.net/Squassh): joint deconvolution-segmentation procedure
Commercial software
- SVI Huygens
- MC AutoquantX
-…
Examples
Original
Original
Original
PID (WPL, Wiener Gamma 0.1,
50IT, bead PSF)
PID (WPL, 50IT, bead PSF)
PID (WPL, 50IT, true PSF)
+ Free & Open source & full control + Reasonably fast + Support for spatially-­‐variant PSF (un-­‐tested) -­‐ High memory usage -­‐ Visually less crispy AutoquantX (30IT, bead
PSF)
AutoquantX (30IT, bead
PSF)
AutoquantX (30IT, true PSF)
+ Fast convergence + Robust algorithms + Very simple to use + Visually appealing results + 2D mode for thin samples -­‐ Expensive & Closed source Huygens (50IT, bead distilled
PSF)
Huygens (50IT, bead distilled
PSF)
Huygens (50IT, distilled true
PSF)
+ Microscope specific PSF + depth-­‐varying PSF + supports spinning disk M. + Visually appealing results -­‐ Expensive & Closed source Courtesy of S. Tosi
Summary
Deconvolution is a computational technique allowing to (partly)
compensate for the image distortion created by an optical system
Correct deconvolution should improve:
attenuation of the out of focus light
quantitative measurements
the spatial resolution
Incorrect deconvolution could:
Introduce (more) artifacts -> reduce image quality
It works best for thin (<50 um), optically transparent, fixed, bright
samples.
Challenging for live microscopy: short exposure (limit motion blur),
objective adapted to medium (limit spherical aberrations).
References
Reviews (overviews):
1.
Waters, Accuracy and precision in quantitative fluorescence microscopy, JCB 2009
2.
Parton et al., Lifting the fog: Image restoration by deconvolution, Cell biology 2006
3.
Pawley, Chapter 25: “Image enhancement by deconvolution”, Handbook of biological confocal
microscopy, 2006
4.
McNally et al., Three-Dimensional Imaging by Deconvolution Microscopy, Methods 1999
Technical articles:
1.
Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy,
Scientific Reports 2013
2.
Bertero et al., Image deconvolution, Proc. NATO A.S.I. 2004
3.
Thiébaut, Introduction to image reconstruction and inverse problems, Proc. NATO A.S.I. 2002
Websites:
1.
Olympus microscopy center (overview):
http://www.olympusmicro.com/primer/digitalimaging/deconvolution/deconvolutionhome.html
2.
Textbook: http://blogs.qub.ac.uk/ccbg/fluorescence-image-analysis-intro
3.
http://fiji.sc/Deconvolution_tips
Thank You!
Slides courtesy:
Pete Bankhead
Queen’s University
Belfast
Sébastien Tosi
IRB Barcelona