Superresolution in Fluorescence and
Diffraction Microscopies with Multiple
Illuminations
- Jules Girard -
2 December 2011
Detector
Introduction : Imaging with optics and resolution
Imaging
device
Parameter of
interest
Probing
function
𝑀 = ( 𝑓𝑜𝑏𝑗 × 𝑓𝑖𝑙𝑙 ) ∗ 𝑝𝑠𝑓
𝑓𝑜𝑏𝑗 × 𝑓𝑖𝑙𝑙
𝑓𝑜𝑏𝑗 ∗ 𝑓𝑖𝑙𝑙
FT
𝑀 = 𝑓𝑜𝑏𝑗 ∗ 𝑓𝑖𝑙𝑙 × 𝑝𝑠𝑓
Low-pass filter
𝑀
𝑀
× 𝑝𝑠𝑓
1/27
Introduction : Extend resolution with illumination
𝑀 = 𝑓𝑜𝑏𝑗 × 𝑓𝑖𝑙𝑙 ∗ 𝑝𝑠𝑓
𝑀 = 𝑓𝑜𝑏𝑗 ∗ 𝑓𝑖𝑙𝑙 × 𝑝𝑠𝑓
kx
ky
𝑓𝑜𝑏𝑗
kx
kx
∗
ky
𝑓𝑖𝑙𝑙
ky
𝑓𝑜𝑏𝑗 ∗ 𝑓𝑖𝑙𝑙 × 𝑝𝑠𝑓
By using multiple and inhomogeneous illuminations, we can shift high frequency parts of
the object spatial spectrum into the passband defined by the psf
W. Lukosz and M. Marchand, Optica Acta 10, 241-255 (1963).
W. Lukosz, JOSA 56, 1463 (1966).
More generally : 𝐦𝐚𝐱(𝒌𝒐𝒃𝒋 ) = 𝐦𝐚𝐱(𝒌𝒑𝒔𝒇 ) + 𝐦𝐚𝐱(𝒌𝒊𝒍𝒍)
2/27
Introduction : Reconstruct a super-resolution image
𝑀𝑖 = 𝑓𝑜𝑏𝑗 × 𝑓𝑖𝑙𝑙,𝑖 ∗ 𝑝𝑠𝑓
𝑀𝑖 = 𝑓𝑜𝑏𝑗 ∗ 𝑓𝑖𝑙𝑙,𝑖 × 𝑝𝑠𝑓
(𝑖 = 1. . 𝑁)
 Inversion → numerical data processing
2 cases
𝑓𝑖𝑙𝑙 is known
 « Direct » inversion
with analytical approach
𝑓𝑖𝑙𝑙 is unknown
 Non-linear inversion
 Find both 𝑓𝑜𝑏𝑗 and 𝑓𝑖𝑙𝑙
with the use of constraints
3/27
Presentation Outline
𝑀 = 𝑓𝑜𝑏𝑗 × 𝑓𝑖𝑙𝑙 ∗ 𝑝𝑠𝑓
𝑀 = 𝑓𝑜𝑏𝑗 ∗ 𝑓𝑖𝑙𝑙 × 𝑝𝑠𝑓
𝐦𝐚𝐱(𝒌𝒐𝒃𝒋 ) = 𝐦𝐚𝐱(𝒌𝒑𝒔𝒇 ) + 𝐦𝐚𝐱(𝒌𝒊𝒍𝒍 )
I.
Optical Diffraction Tomography
II.
Structured Illumination Fluorescence Microscopy
4/27
I. Optical Diffraction Tomography
II. Optical Diffraction Tomography
𝑥
Fourier Space
= 0 for lateral
frequencies
> 𝑘0 𝑁𝐴
We measure :
𝑘
𝐸𝑑 (𝑘) = 𝑀 = 𝑓𝑜𝑏𝑗 𝑘 ∗ 𝑓𝑖𝑙𝑙 𝑘
𝜽𝒊𝒏𝒄
𝑧
(Sample dielectric
permittivity contrast)
Δ𝜀
× 𝑝𝑠𝑓(𝑘)
internal
Etot (Total
electric field)
Objective
𝐸𝑡𝑜𝑡 𝑥, 𝑦, 𝑧 = 𝐸𝑟𝑒𝑓 𝑥, 𝑦, 𝑧 + 𝐸𝑑 𝑥, 𝑦, 𝑧
 Reconstruct 𝜀(𝑥, 𝑦, 𝑧) : quantitative microscopy of unstained sample
 ≠ illuminations → ≠ Etot → access to ≠ parts of Δ𝜀
E Wolf, Optics Communications 1, 153-156 (1969).
V Lauer, Journal of Microscopy 205, 165-76 (2002).
5/27
II. Optical Diffraction Tomography
Experiment
 Illumination with « plane waves » under ≠ incidences
 Measure complex values of 𝐸𝑑 (𝑘𝑥 )
(G. Maire, F. Drsek, H.Giovannini)
Laser
(λ=633nm)
Phase
modulator
 Calibration and normalization
CCD
𝑀 → 𝐸𝑑 (𝑘𝑥 )
 Inversion : 𝐸𝑑 (𝑘𝑥 ) → Δ𝜀(𝑥, 𝑧)
Sample
6/27
II. Optical Diffraction Tomography
1. 𝐸𝑡𝑜𝑡 𝑥, 𝑦, 𝑧 = 𝐸𝑟𝑒𝑓 𝑥, 𝑦, 𝑧 + 𝐸𝑑 𝑥, 𝑦, 𝑧
2. 𝐸𝑑 = 𝑓𝑜𝑏𝑗 ∗ 𝑓𝑖𝑙𝑙 × 𝑝𝑠𝑓
 Low Δ𝜀 : Born Approximation
Δ𝜀
Etot
 𝑓𝑖𝑙𝑙 = 𝐸𝑟𝑒𝑓
 𝐸𝑟𝑒𝑓 is diffraction limited → Abbe limit
𝐦𝐚𝐱(𝒌𝒐𝒃𝒋 ) = 𝐦𝐚𝐱(𝒌𝒑𝒔𝒇 ) + 𝐦𝐚𝐱(𝒌𝒊𝒍𝒍 )
𝒅 = 𝝀/(𝟐𝑵𝑨)
𝟐𝒌𝟎 𝑵𝑨
𝒌𝟎 𝑵𝑨
𝒌𝟎 𝑵𝑨
 High Δ𝜀 : Multiple Scattering Regime
 𝐸𝑡𝑜𝑡 depends on object and illumination
 𝐸𝑡𝑜𝑡 is not diffraction limited → resolution improvement ? (𝐦𝐚𝐱(𝒌𝒊𝒍𝒍 ) > 𝒌𝟎 𝑵𝑨 ?)
7/27
II. Optical Diffraction Tomography
𝐸𝑡𝑜𝑡 simulations
air
• λ = 633 nm
z
50 nm
25 nm
x
50 nm
glass
• Abbe limit with NA = 1.5
→ 211 nm
50°
Low 𝛥𝜀
Δ𝜀 = 10-2
High Δ𝜀 (Ge)
𝐸𝑡𝑜𝑡
Δ𝜀 = 28.8
𝐸𝑡𝑜𝑡
< 𝝀/𝟔
𝐦𝐚𝐱(𝒌𝒊𝒍𝒍 ) > 𝒌𝟎 𝑵𝑨 !
8/27
II. Optical Diffraction Tomography
Experimental validation
air
z
50 nm
 Germanium rods
 TIRF configuration (10 angles)
25 nm
50 nm
glass
x
 NA = 1.3
→ Abbe limit : 245 nm
(A. Talneau – LPN)
Z (µm)
0,5
0
Z (µm)
0,5
0
11
9/27
II. Optical Diffraction Tomography
Conclusion
 We achieved quantitative reconstruction of the permittivity map
of unstained sample even with a multiple scattering regime
 Multiple scattering : drawback way to improve the resolution of
ODT far beyond diffraction limit
10/27
II. Structured Illumination in
Fluorescence microscopy on 2D samples
CCD
III. Structured Illumination Microscopy in Fluorescence
Objective
Tube
Lense
𝑀 = 𝑓𝑜𝑏𝑗 × 𝑓𝑖𝑙𝑙 ∗ 𝑝𝑠𝑓
𝜌
𝐼
(fluorescence
density)
(field intensity)
𝑘𝑦
1
0,5
𝑘𝑥
0
𝑝𝑠𝑓
−2𝑘0 𝑁𝐴
(2D)
𝑝𝑠𝑓
(2D and 1D)
+2𝑘0 𝑁𝐴
𝑘𝑥
11/27
III. Structured Illumination Microscopy in Fluorescence
 Use periodic pattern → 𝐼 ∝ 1 + cos 𝐾 ⋅ 𝑟 + Φ
𝑀 = 𝑓𝑜𝑏𝑗 × 𝑓𝑖𝑙𝑙 ∗ 𝑝𝑠𝑓
𝜌
R. Heintzmann and C. Cremer, SPIE, pp. 185-196. (1998)
𝐼
Mats G L Gustafsson, Journal of Microscopy 198, 82-7 (2000).
kx
𝐾
∗
ky
𝑓𝑜𝑏𝑗
=
𝑓𝑖𝑙𝑙
𝑀
 Requirements for illumination pattern :
• Accurate translation → needed for discrimination of the 3
copies 𝑓𝑜𝑏𝑗
• High contrast → higher SNR (no dim for shifted copies of 𝑓𝑜𝑏𝑗 )
12/27
III. Structured Illumination Microscopy in Fluorescence
Limit : Illumination pattern is diffraction limited : 𝐦𝐚𝐱(𝒌𝐢𝐥𝐥 ) ≤ 2𝑘0 𝑁𝐴
 𝐦𝐚𝐱(𝒌𝒐𝒃𝒋 ) = 𝟒𝒌𝟎 𝑵𝑨 : twice better than classical WF
 How can we reach higher frequencies ?
 Use of non-linearities : 𝑓ill = 𝐹 𝐼 → M = 𝜌 × 𝐹(𝐼) ∗ 𝑝𝑠𝑓
 𝐦𝐚𝐱(𝒌𝑖𝑙𝑙 ) > 2𝑘0 𝑁𝐴
(R. Heintzmann et al., JOSA A, 19, 2002 & M G L Gustafsson, PNAS, 102, 2005)
 Get 𝑘I below diffraction limit (surface imaging)
 High index substrate
→ limited n and/or absorption
 Nanostructured devices with plasmonics
→ field bound to the structure + difficulties to cover a large area
13/27
III. Structured Illumination Microscopy in Fluorescence
Grating assisted Structured Illumination Microscopy

Dielectric resonant grating ≈ 2D waveguide + 2D sub-λ grating
𝑧
𝒌𝒎𝒐𝒅𝒆
Glass coverslip
𝑛 ≈ 1.515 @ 633nm

a-Si layer
𝑛 ≈ 4 + 0.1 𝑖 @ 633nm
𝒌𝒊𝒏𝒄
Hexagonal geometry : 6 equivalent orientations → near isotropic resolution
𝑘𝑥
𝒙
𝒚
𝑘𝑦
z=0

Design optimization → numerical simulations
14/27
III. Structured Illumination Microscopy in Fluorescence
Gratings fabrication process
(J. Girard, A. Talneau, A. Cattoni LPN – CNRS)
1. aSi deposition (PECVD)
2. Grating patterning
(e-beam + RIE)
3. Planarization
(A. Cattoni)
A. Cattoni, A. Talneau, A-M Haghiri-Gosnet, J. Girard, A. Sentenac (oral presentation, MNE 2011)
15/27
III. Structured Illumination Microscopy in Fluorescence
Excitation modes of the grating substrate
2 beams excitation
1 beam excitation
𝒌𝒎𝒐𝒅𝒆
𝒌𝒎𝒐𝒅𝒆
𝒌𝒊𝒏𝒄
left
𝒌𝒊𝒏𝒄
right
𝐼(𝑥, 𝑧 = 0) ∝ 1 + 𝐴 cos(𝑲−𝟏,𝟎 ⋅ 𝑟∥ + 𝜙𝑖 )
𝐾−1,0 ≈ 1.3 × (2𝑘𝑂 𝑁𝐴)
𝐼 ∝ 1 + 𝐴 cos 𝑲−𝟏,𝟎 ⋅ 𝑟∥
+ 𝐵 cos 𝟐𝒌𝒊𝒏𝒄 ∥ ⋅ 𝑟∥ − 𝜑
+ 𝐶 cos 𝟐 𝑲−𝟏,𝟎 + 𝒌𝒊𝒏𝒄 ∥ ⋅ 𝑟∥ − 𝜑
+ 𝐷 cos( 𝑲−𝟏,𝟎 + 𝟐𝒌𝒊𝒏𝒄 ∥ ⋅ 𝑟∥ − 𝜑)
𝐾−1,0 + 2𝑘𝑖𝑛𝑐 ∥ < 2𝑘𝑂 𝑁𝐴
2 𝐾−1,0 + 𝑘𝑖𝑛𝑐 ∥ ≈ 1.6 × 2𝑘𝑂 𝑁𝐴
17/27
III. Structured Illumination Microscopy in Fluorescence
Experimental setup
 Control of orientation, phase
and incidence angle on the
substrate (65°)
16/27
III. Structured Illumination Microscopy in Fluorescence
Grating characterization : SNOM measurements
(Geoffroy Scherrer, ICB, Dijon)
High Frequency Pattern from the Grating
Stretched fiber
65°
z
=
Grid Shifting
1 beam excitation
𝒌𝒎𝒐𝒅𝒆
𝒌𝒊𝒏𝒄
𝒌𝒎𝒐𝒅𝒆
𝒌𝒊𝒏𝒄
simulation
Theoretical
simulation
18/27
III. Structured Illumination Microscopy in Fluorescence
Grating characterization : Far field fluorescence measurements
 2 beams excitation : Low frequency component of the intensity pattern
𝐼 ∝ 1 + 𝐴 cos 𝑲−𝟏,𝟎 ⋅ 𝑟∥
+ 𝐵 cos 𝟐𝒌𝒊𝒏𝒄 ∥ ⋅ 𝑟∥ − 𝜑
+ 𝐶 cos 𝟐 𝑲−𝟏,𝟎 + 𝒌𝒊𝒏𝒄 ∥ ⋅ 𝑟∥ − 𝜑
+ 𝐷 cos( 𝑲−𝟏,𝟎 + 𝟐𝒌𝒊𝒏𝒄 ∥ ⋅ 𝑟∥ − 𝜑)
 WF Fluorescence observation
with ~homogeneous layer of fluorescent beads
19/27
III. Structured Illumination Microscopy in Fluorescence
 Our manufactured gratings can produce a grid of light with
 180 nm period (λ/3.5) (down to 147 nm, λ/4.3 with alternative design)
 a high contrast
 The possibility to shift its position
 According to
𝐦𝐚𝐱(𝒌𝒐𝒃𝒋 ) = 𝐦𝐚𝐱(𝒌𝒑𝒔𝒇 ) + 𝐦𝐚𝐱(𝒌𝒊𝒍𝒍 ) , a final resolution
of up to 87 nm could be reached at λ =633 nm!
 However we need to know the illumination pattern for inversion procedure
20/27
III. Structured Illumination Microscopy in Fluorescence
“Blind” SIM Inversion
M1 = 𝐼1 × 𝜌 ∗ 𝑃𝑆𝐹
M2 = 𝐼2 × 𝜌 ∗ 𝑃𝑆𝐹
…
Mn = 𝐼𝑛 × 𝜌 ∗ 𝑃𝑆𝐹
1
𝑁
𝑁
𝑵 + 𝟏 unknowns :
𝐼𝑛 = 𝐼0
𝜌 + 𝑁 × 𝐼𝑖
𝑖=1
𝑵 equations
+1
 Iterative optimization of estimates of 𝐼𝑖 and 𝜌
(Emeric Mudry & Kamal Belkebir)
through minimization of a cost function :
𝑁
F 𝜌, 𝐼1 , … , In =
𝑀𝑖 − 𝐼𝑖 × 𝜌 ∗ 𝑃𝑆𝐹
𝑖=1
2
24
21/27
III. Structured Illumination Microscopy in Fluorescence
Experimental validation
 Observation of fluorescent beads (Ø 90nm) immersed in glycerin with
classical SIM
WF image
Deconvolution of the WF image
Our Result
Optimized « analytical » algorithm
Inversion by Pr. R. Heintzmann
22/27
III. Structured Illumination Microscopy in Fluorescence
Speckle illumination
 Speckle pattern is a perfect candidate for SIM with our ‘blind’ inversion
algorithm
1. Contains every accessible
frequencies
Simulation
𝐼
2. Known average
illumination
3. Experiment far simpler
than standard SIM
Measurement
1
𝑁
𝑁
𝐼𝑛 (𝑥, 𝑦)
𝑁→∞
𝐼0
𝑖=1
𝐼
23/27
III. Structured Illumination Microscopy in Fluorescence
Speckle illumination : simulations
object
WF image
One measured image
Deconvolution
speckles
0,5𝑁𝐴𝑑𝑒𝑡
speckles
1 𝑁𝐴𝑑𝑒𝑡
Deconvolution
𝑁𝐴𝑖𝑙𝑙 = 0,5𝑁𝐴𝑑𝑒𝑡
𝑁𝐴𝑖𝑙𝑙 = 1𝑁𝐴𝑑𝑒𝑡
× N ≈ 80
Photon budget :
average of 130
photon/pixel/image
Reconstructed 𝜌
𝐦𝐚𝐱(𝒌𝒐𝒃𝒋 ) = 𝐦𝐚𝐱(𝒌𝒑𝒔𝒇 ) + 𝐦𝐚𝐱(𝒌𝒊𝒍𝒍 )
24/27
III. Structured Illumination Microscopy in Fluorescence
Speckle illumination : experimental results
Rabbit Jejunum slices (150nm thick) (Cendrine Nicoletti, ISM, Marseille)
TEM image of a similar sample
WF image
Reconstructed image
from 100 speckle illuminations
Deconvolution of WF image
25/27
General Perspectives
I. Optical Diffraction Tomography :
 Extend to 3D samples
 Use other configuration (grating substrate, mirror substrate…)
II. Structured Illumination in Fluorescence Microscopy
1. Grating-assisted SIM :
 Make super-resolved images of real samples : use a priori information for
inversion procedure
2. Speckle illumination :
 Extend to 3D samples
27/27
III. Structured Illumination Microscopy in Fluorescence
Conclusion
 SIM with unknown illumination patterns
 Extension of SIM to the use of random speckle patterns
 Not effective yet for grating-assisted SIM (inhomogeneous average
illumination)
26/27
Thanks…
Eric Le Moal
Guillaume Maire
Emeric Mudry
Kamal Belkebir
Anne Sentenac



Geoffroy Scherrer
Anne Talneau
Andrea Cattoni

The whole MOSAIC team for advices, seminars, discussions, equipment, facilities…
Thank you for your attention
II. Optical Diffraction Tomography
z
air
 NA = 0.7 (used up to 0.53 only for illumination)
→ Abbe limit : 500 nm (450nm for full NA)
100 nm
110
300 nm
x
AFM profile
Si
Reconstructed 𝜀 map
Reconstructed 𝜀 profile
Reconstructed 𝜀 profile
(linear inversion)
33
II. Optical Diffraction Tomography
Multiple scattering and resolution
𝐸𝑡𝑜𝑡
Simulation of 𝐸𝑑 (𝛼) = 𝐸𝑑 (𝜃𝑑𝑒𝑡 )
for a plane wave illumination (incidence 50°)
100nm
Δ𝜀 = 10-2
25nm
Δ𝜀 = 28.8
(Germanium)
(a) 𝛥𝜀𝑎𝑣𝑔 = 2 (b) 𝛥𝜀𝑎𝑣𝑔 = 7 (c) 𝛥𝜀𝑎𝑣𝑔 = 14
Modulation of 𝛥𝜀 for the object 2 : 5𝑘0 (>
2𝑘0 𝑁𝐴)
34
III. Structured Illumination Microscopy in Fluorescence
Grating assisted SIM : getting some images
 Problem with inversion : Intensity pattern is not perfectly known
 Speckle algorithm is not able to retrieve frequencies > 2𝑘0 𝑁𝐴
 Add of a priori information (rough orientation and frequencies)
35