Leica - CNIC 1st Practical School in Super-Resolution Microscopy, Centro Nacional de Investigaciones Cardiovasculares Carlos III (CNIC) Image Restoration by Deconvolution: Concepts and Applications Chong Zhang SIMBioSys, Depertment of Information and Communication Technologies Universitat Pompeu Fabra 15th March, 2016 What is deconvolution in your mind? Convolution Deconvolution Point Spread Function Noise Fourier Transform Spatial resolution Pixel size Rayleigh Criterion Airy disk Numerial Aperture Refractive Index Wavelength How do you perform deconvolution? Fiji plugins Huygens Professional MC AutoquantX … What is deconvolution (in microscopy)? Deconvolution is a computational technique allowing to partly compensate for the image distortion caused by a microscope. The betterment can be significant both in terms of attenuation of the out of focus light and increase of the spatial resolution. It was first devised at the MIT for seismology (Robinson, Wiener, early 50’), then applied to astronomy and finally found its way to 3D optical fluorescence microscopy (Agard 1984). It should not be seen as a “black box” to enhance image quality since it can introduce artifacts or further degrade low quality images. It is compatible with quantitative measurements (should even improve). It works best for thin (<50 um), optically transparent, fixed, bright samples. Challenging for live microscopy: short exposure (limit motion blur), objective adapted to medium (limit spherical aberrations). Courtesy of S. Tosi Ideal case Image Detector Ideal case Objective Specimen Courtesy of P. Bankhead Reality Image - Blurred Detector More realis-c Objective Specimen Courtesy of P. Bankhead Reality Image - Blurred - Noisy Detector More realis-c Objective Specimen Courtesy of P. Bankhead Noises along the optical train in digital microscopy Courtesy of S. Tosi Convolution Convolution consists of replacing each point in the original object with its blurred image in all dimensions and summing together overlapping contributions from adjacent points to generate the resulting image Filter kernel !" Original Filtered image image Originalimage image Filtered “stamping” the kernel on each pixel of the image the kernel is scaled (multiplied) by the intensity of the central pixel Accumulated (summed) in the output image. Courtesy of S. Tosi Convolution Convolution consists of replacing each point in the original object with its blurred image in all dimensions and summing together overlapping contributions from adjacent points to generate the resulting image = = A 3D kernel is a stack holding the filter coeﬃcients Courtesy of S. Tosi Fourier Transform (FT) vs convolution If we take the FT of the equation, the is replaced by multiplication, thus image restoration might be achievable by: dividing the FT of the image by the FT of the kernel and then taking the inverse Fourier transform. = Spatial Domain Fourier Domain X = Courtesy of S. Tosi Point Spread Function (PSF) A record of how much the image created by a microscope spreads/blurs an object of a single point (thus determines the way in which images of objects blur into each other in the final image). Widefield PSF Bankhead 2014 Cannell 2006 Theoretical PSF vs Measured PSF Measured PSF: an image resulting from a single small spherical fluorescent bead (smaller than the optical resolution, thus forms eﬀectively a point source of light) Theoretical PSF: generated by a computer program, after input of values describing the optical system – the magnification, NA of the objective, the illuminating and emitted wavelengths, and the refractive index of the objective lens immersion medium. It gives an indication of the best possible resolution for a given objective but these limits are not achievable. Real PSFs are typically >20% bigger than calculated versions. Measured PSF (experimental PSF) Parton 2006 Theoretical PSF Bankhead 2014 Courtesy of S. Tosi Gated 3D STED STED /GS <50 <130 70 70 560 <130 Courtesy of N. Garin PSF in the focal plane (where most things are measured) Airy disk Spatial resolution: radius of the smallest point source in the image, i.e. first minimum of Airy disk Parton 2006 Bankhead 2014 Widefield PSF: 100nm beads, excitation 520nm, emission 617nm Rayleigh criterion λ : fluorophore emission wavelength NA : objective numerical aperture n : refractive index of the objective lens immersion medium NA = n sinθ NA can never exceed n, which itself has fixed values (e.g. 1.0 for air, 1.33 for water, or 1.52 for oil) Bankhead 2014 Notes: 1. Rayleigh criterion has not taken into account the eﬀects of: brightness, pixel size, noise 2. High NAs are possible when the immersion refractive index is high What else does measured PSF tell us? Asymmetry radial (x-y): commonly misalignment of optical components about the z-axis, either as tilt or decentration along the optical axis (z-axis): commonly due to spherical aberration, which may result from refractive index mismatches between the objective, immersion medium, and sample or tube length/coverslip thickness errors. Pawley 2006 Notes: 1. The immersion refractive index should match the refractive index of the medium surrounding the sample, to avoid spherical aberration 2. Item 1 is often strongly preferable to using the highest NA objective available, as it is usually better to have a larger PSF than a highly irregular one. Deconvolution principle Noise Digital filter Microscope PSF Deconvolved image Object (sample) Courtesy of S. Tosi =R =D The deconvolution filter F should “undo” the eﬀect of the microscope PSF H by processing the sampled image R, ideally D = S. Deconvolution principle Noise Digital filter Microscope PSF Deconvolved image Object (sample) Courtesy of S. Tosi =R =D The deconvolution filter F should “undo” the eﬀect of the microscope PSF H by processing the sampled image R, ideally D = S. Classification of deconvolution methods Nearest neighbors Deblurring subtractive No neighbors Fast Software available Not for 2D only quantitative No need PSF intensity measures Tikhonov-Miller filter Linear inverse Regularized inverse Fast 3D Software available Needs PSF Wiener filter Jansson van Cittert 3D Needs PSF Constrained iterative Richardson-Lucy Lifting the Fog: Image Restoration by Deconvolution, Parton 2006 3D Estimates PSF Not for super resolution Trade-oﬀ between sharpness & noise Use noise models Slow Good 3D results Adaptive blind Do not count for noise May not be always quantitative Also for super resolution Linear deconvolution: inverse filter deconvolution For example, assuming H known, F linear (convolution) and no noise (N = 0) leads to: But in practice… Noise enhancement ruins our eﬀorts! 2,4·∙107 4,1·∙10-‐8 1 A very simple model for the PSF H (Gaussian std = 1 pixel) H power spectrum (log display) overlaid with raw values 1 H-1 power spectrum (log display) overlaid with raw values Courtesy of S. Tosi Inverse filter deconvolution H is a Gaussian with std = 2 pixels +N H H-1 Original image S Noise std = 10-4 S after convolution by H Noise std = 10-12 No noise Courtesy of S. Tosi Second try: regularized inverse 4,1·∙10-‐8 100 7 2,4·∙10 1 A very simple model for the PSF H (Gaussian std = 1 pixel) H power spectrum (log display) overlaid with raw values 1 (H-1)reg (1% clipping) power spectrum (log display) overlaid with raw values Courtesy of S. Tosi Regularized Inverse Filter Deconvolution +N H Original image S S after convolution by H (H-1)trunc Noise std = 10-4 Courtesy of S. Tosi Third Try: Wiener Filter The Golden Linear Deconvolution Trade-oﬀ Coming back to: Minimizing the expectation of ||E|| over all possible noise realizations assuming a white Gaussian noise: Wiener filter Bands free of noise: |N(u,v)| = 0 ! F(u,v) = H(u,v)-1 (inverse filter) Strong noise bands: |N(u,v)|! ∞ ! F(u,v) ! 0 (cut-oﬀ) Intermediate bands: best trade-oﬀ As noise at certain frequencies increases, the signal-‐to-‐noise ra-o drops, so F also drops. This means that the Wiener filter attenuates frequencies dependent on their signal-to-noise ratio. Courtesy of S. Tosi Wiener Deconvolution Regularized inverse filter result Noise std = 10-4 Wiener filter result Noise std = 10-4 Courtesy of S. Tosi Non-Linear Deconvolution The best deconvolution algorithms for 3D microscopy are typically non-linear. Principle of Maximum A Priori algorithms (MAP): The second equality comes from Bayes theorem. In the optimization S is usually constrained to be positive and somehow spatially smooth (TV regularization term) " Pr(S). The statistical distribution of the noise has to be known to derive the maximum likelihood term Pr(R|S) " the algorithm is tuned to a particular noise (e.g. Poisson or Gaussian noise). There is usually no known analytical solution to the problem, the algorithms proceeds by iterations (candidate Si at iteration i) to refine the estimate of the data at each iteration. The Richardson-Lucy algorithm is among the most well known MAP deconvolution algorithm. Some algorithms also simultaneously estimate the PSF from the sampled image (blind deconvolution). Courtesy of S. Tosi Quantification after deconvolution Ideally: relocate signal to the point of origin in 3D, thus conserve the sum of fluorescence signal. It improves quantification! In practice: diﬀerent algorithms have more or less compromises Quantitative intensity measurements, e.g. intensity ratio: controls, also report on un-deconvolved data for comparison Quantitative positional or structural analysis, e.g. centroid, tracking, volume analysis, (object based) colocalisation, etc: relatively less critical the choice For all analysis: Deconvolution process comparable between datasets Compare with control/un-deconvolved data Understand algorithm used and choose most suitable Report possible artifacts and confirm it, if possible Deconvolution tools Fiji plugins Diﬀraction PSF 3D PSF generator Parallel iterative deconvolution (http://imagej.net/Parallel_Iterative_Deconvolution): 4 deconvolution algorithms Parallel spectral deconvolution (http://imagej.net/Parallel_Spectral_Deconvolution): not iterative, no constraint e.g. nonnegativity Iterative Deconvolve 3D (http://imagej.net/Iterative_Deconvolve_3D): non-negative, iterative, similar to WPL algorithm. The execution is way slower on modern (multicore) computers but the memory requirement is less stringent DeconvolutionLab (http://bigwww.epfl.ch/algorithms/deconvolutionlab/): diﬀerent algorithms including a custom version of the thresholded Landweber algorithm Squassh (http://imagej.net/Squassh): joint deconvolution-segmentation procedure Commercial software - SVI Huygens - MC AutoquantX -… Examples Original Original Original PID (WPL, Wiener Gamma 0.1, 50IT, bead PSF) PID (WPL, 50IT, bead PSF) PID (WPL, 50IT, true PSF) + Free & Open source & full control + Reasonably fast + Support for spatially-‐variant PSF (un-‐tested) -‐ High memory usage -‐ Visually less crispy AutoquantX (30IT, bead PSF) AutoquantX (30IT, bead PSF) AutoquantX (30IT, true PSF) + Fast convergence + Robust algorithms + Very simple to use + Visually appealing results + 2D mode for thin samples -‐ Expensive & Closed source Huygens (50IT, bead distilled PSF) Huygens (50IT, bead distilled PSF) Huygens (50IT, distilled true PSF) + Microscope specific PSF + depth-‐varying PSF + supports spinning disk M. + Visually appealing results -‐ Expensive & Closed source Courtesy of S. Tosi Summary Deconvolution is a computational technique allowing to (partly) compensate for the image distortion created by an optical system Correct deconvolution should improve: attenuation of the out of focus light quantitative measurements the spatial resolution Incorrect deconvolution could: Introduce (more) artifacts -> reduce image quality It works best for thin (<50 um), optically transparent, fixed, bright samples. Challenging for live microscopy: short exposure (limit motion blur), objective adapted to medium (limit spherical aberrations). References Reviews (overviews): 1. Waters, Accuracy and precision in quantitative fluorescence microscopy, JCB 2009 2. Parton et al., Lifting the fog: Image restoration by deconvolution, Cell biology 2006 3. Pawley, Chapter 25: “Image enhancement by deconvolution”, Handbook of biological confocal microscopy, 2006 4. McNally et al., Three-Dimensional Imaging by Deconvolution Microscopy, Methods 1999 Technical articles: 1. Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy, Scientific Reports 2013 2. Bertero et al., Image deconvolution, Proc. NATO A.S.I. 2004 3. Thiébaut, Introduction to image reconstruction and inverse problems, Proc. NATO A.S.I. 2002 Websites: 1. Olympus microscopy center (overview): http://www.olympusmicro.com/primer/digitalimaging/deconvolution/deconvolutionhome.html 2. Textbook: http://blogs.qub.ac.uk/ccbg/fluorescence-image-analysis-intro 3. http://fiji.sc/Deconvolution_tips Thank You! Slides courtesy: Pete Bankhead Queen’s University Belfast Sébastien Tosi IRB Barcelona