3D fluorescence microscopy
For 3D, acquire a “focal series” (stack) of images:
Take an image, refocus the sample, take another image, refocus, etc.
Problem: Each image contains out-of-focus blur from other focal planes
Approach 1:
Physically exclude the blur by
confocal microscopy
Approach 2:
Remove the blur computationally
Light
detector
Detection
pinhole
Stack of 2D images
Laser
Processing
(deconvolution)
Excitation
pinhole
3D reconstruction
Scan sample or beam
to gather a 3D data set
Sample
How well can this be done?
Deconvolution of 3D data
(Dividing fission yeast cell)
Raw data
Deconvolved
The Fourier Transform
Any (nice) function g(x) can
be equally well described as
a sum of waves.
The “Fourier transform”
~
g(k) specifies the amplitude A and
the phase φ for the component wave
of wavelength L = 1 / k
Long wavelength (low resolution)
info is close to the origin
~
g(k)
k
Fourier Transform and Digital Filter Applets
http://www.falstad.com/mathphysics.html
Low Resolution Duck
A Duck
a blurred duck
and its scattering function (Fourier Transform):
<=
Using only low resolution data
Convolutions
(f
g)(r) =
f(a) g(r-a) da
Why do we care?
- They are everywhere…
- The convolution theorem:
If
h(r) = (f
g)(r),
A convolution in real space becomes
a product in reciprocal space & vice versa
h(k) = f(k) g(k)
then
Symmetry: g
So what is a convolution, intuitively?
- “Blurring”
- “Drag and stamp”
f
f
g
g
=
y
y
x
y
x
=
x
f = f
g
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from arjournals.annualreviews.org
by University of California - San Francisco on 05/09/10. For personal use only.
""
2D PSF, OTF of an in-focus lens "
viewing
a point
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from
by University of California - San Francisco on 05/09/10. F
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from arjournals.annualreviews.org
by University of California - San Francisco on 05/09/10. For personal use only.
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from arjournals.annual
by University of California - San Francisco on 05/09/10. For personal use o
).+. '3̸@̸6̸̸'̸̸32@61̸66@%J̸33@61̸'%ǁ̸
Ýnn̸
LOW:,%AÒ T/,V9LIÒ G9,SLT,LO^Ò
őƕŕ‚ǵhr̸H® ̸|rÕ2àhrºċP.H̸‚ȏ̸¸]̸̸.!]H̸H̸̸HC#.RJ̸
K33@lCɾ3̸@l6̸1@̸æ'3̸̸32̸36̸@3@@J̸
̸;_˜RɃi̸ƒl̷̸
$*
+.
('-'. &. (.
)+
̸a̸̸#.]]̸
b\
frhr0n*r r ™p̸
H]̸
}sÕ1à
hraH̸̸C#C#.̸’db̸
~#̸Ffrhrjr̸≠'̸jr ǵ ≠ ̸jr/r dr̸a̸̸̸'̸8̸ƀ̸ɫŒ̸
661̸33%̸3K61̸66̸@̸̸@@g̸'3̸
̸%23ŀ̸g̴̵̸
8-̸8̸ʍ̸̓H-8ʙ9̸-̸!̸a88̸̸̸Ÿ8̸-89Ŗ̸
X
w̸66̸ ̸32̸ @̸ %l̸ @2̸Pĥ{x1̸3̸xìx̸33%̸K3{ìxCʏĥ%̸3̸ʗ3x̸̸x̸6̸˵̸
326@l•g̸ K222̸ $)
.+.2ɡ̸
23̖2̸ˠ23q̸
).+. '3̸@̸6̸̸'̸̸32@61̸66@%J̸
33@61̸'%ǁ̸
XfhSk[r ǵ
G 6h jrrafh j\jr ?¡r
'(r
¸]̸
0nr ]]̸ ̸ H9̸#̸̸ !¾̸ ®̸ #]̸ ̸ #$̸
d̸̸Ɍ33xʵC'3̸1@̸{̸{̸@̸K̸{ēɿ3@{̸6{Ïx%i̸ƒ@̸ÏJ̸
ȯ̸
K33@lCɾ3̸@l6̸1@̸æ'3̸̸32̸36̸@3@@J̸
̸;_˜RɃi̸ƒl̷̸
®*̸cɋïï̸ċP.H̸‚̸
̸2X
'̸@̸31̸'ƌ̸̸K‰2̸32%̸K1̸@̸6̸66̸32̸@2̸2̸6˖ŕ̸
a̸;!̸'̸8̸ċP8̸ (r ̸̸8̸!̸¾8̸-8̸98̸
661̸33%̸3K61̸66̸@̸̸@@g̸'3̸
̸%23ŀ̸g̴̵̸
8ÿ̸N'ĺ̸8̸-̸'̸̸8E̸̸̸8̸89̸†
%@3x%*̸vì̸ĥ1̸%x3x̸3{̸'3̸x̸'x̸@ìx̸x̸ˊ6̸'̸
@̸31Ǯȇ̸
½Õ
Pĥ{x1̸3̸xìx̸33%̸K3{ìxCʏĥ%̸3̸ʗ3x̸̸x̸6̸˵̸
/àhr + Ifr7r¯Õ1àhrrc\frhr0o#r
̸ ̸8̸8̸8Ã ŧ9̸-98ć̸
32̸32%̸K1̸2̸K‰g2̸6Ȑ̸g̸23̸3%Ŀ̸@2̸12̸̸̸ė@ƪȈ̸
23̖2̸ˠ23q̸
b]̸HŒ̸̸a̸ H¶]̸HÃ] ̸#.H̸̸!̸] ̸
326Ï̸3q̸ȥ̸@̸̉S̸̸1l6̸32«̸3x32̸«̸3l%%ǂ̸
d̸̸Ɍ33xʵC'3̸1@̸{̸{̸@̸K̸{ēɿ3@{̸6{Ïx%i̸ƒ@̸ÏJ̸
ȯ̸
U[Aà
Xfhlr ǵ Hfhl afhmr
H̸̸9̸ŞdbJ̸]!̸H9H#H̸H̸PH̸̸
'̸@̸31̸'ƌ̸̸K‰2̸32%̸K1̸@̸6̸66̸32̸@2̸2̸6˖ŕ̸
%̸ ÷2̸ %÷23̸ „̸ ÷̸ K‰2@2̸ 6̸ @6'̸@̸ ̸ ÷2̸ 23@32*̸ ƒɭ̸
a̸ afrhrjr -8̸8̸Ƥ̸889̸8à -9̸8†
%@3x%*̸vì̸ĥ1̸%x3x̸3{̸'3̸x̸'x̸@ìx̸x̸ˊ6̸'̸
@̸31Ǯȇ̸
'&$r
½Õ2à
hr ¨Õ2àhr„ ËÕ2àhr0 ¡X
2x̸6̸H„ ƫ̸ČÏ32̸zR̸Ï̸326@%̸@̸@2̸3l6̸23@32̸'̸@ɮ̸
̸µ-̔-i̸
32̸32%̸K1̸2̸K‰g2̸6Ȑ̸g̸23̸3%Ŀ̸@2̸12̸̸̸ė@ƪȈ̸
6̸K1̸
α
Ⱥ̸̸H
]̸̸HE̸H̸§ÙÕ0à[ǵnr̸!̸̸.̸#̸̸E̸#̸{à
326Ï̸3q̸ȥ̸@̸̉S̸̸1l6̸32«̸3x32̸«̸3l%%ǂ̸
H̸H̸
¨Õ2à
hrnrKr ® ̸̸ H¶]̸HH̸;¡Ƭ̸®̸̸¡ǐZH̸
›eMp̸
ǵ
-ǵ
g̸
I„
%̸ ÷2̸ %÷23̸ „̸ ÷̸ K‰2@2̸ 6̸ @6'̸ @̸ ̸ ÷2̸ 23@32*̸ ƒɭ̸
]
̸!̸̸H̸
0p$r N̸H̸]] ̸#]̸̸H]̸E̸¸H̸
2x̸6̸
H„ ƫ̸ČÏ32̸zR̸Ï̸326@%̸@̸@2̸3l6̸23@32̸'̸@ɮ̸
32̸ -ǵ ̸ ̸
g%ɯy̸ '̸@2̸ 2%g̸ K222̸ 2̸ 6̸ %̸ @ɰ̸
̸#9̸9̸
̸ ̸ Yi„ H̸̸̸.̸#̸̸#̸̸
H 32'3@g̸
HH 9̸9̸
6̸K1̸
2g̸;_ǣ≠'3̸ 3ü̸_qő_ő̸'3̸ ̸g6̸3̸6Rq̸ƒ̸@̸@6̸
α ≠!̸̸]]H̸!®]]H̸#.Hć̸
maximum angle
H]
'3m1̸@@̸̸K̸2%̸K1̸@̸1@̸Ƙ̸«̸
›eMp̸ ǵ -ǵN„g̸ I„
25x, 0.5NA
PSF
''"r@ɰ̸
Zg8Sk[r
Fg
i nNagi
+̸
&
n\n0m!r
G3ì
+ ǵ Ôzǰ̸̸ɇƯÜ̸
Sì 6̸ %̸
32̸
-ǵ
̸
32'3@g̸
g%ɯy̸
'̸@2̸ 2%g̸
K222̸ 2̸
the highest
K,Tr spatial frequency is then
2g̸;_ǣ≠'3̸
3ü̸_qő_ő̸'3̸
fc = 0.178 µm for
λ= 500nm,̸g6̸3̸6Rq̸ƒ̸@̸@6̸
1.4NA oil
immersion
lens
̸
QìK2̸2̸6̸&̸̸6̸2%i̸žg̸Ď62gƦ̸3g23gS̸ǃƗ̸
à̸̸9HH̸̸
0mr„ &r ̸{à
+ ,„ ̸êH̸̸#̸PH̸
''r!̸̇
'3m1̸@@̸̸K̸2%̸K1̸@̸1@̸Ƙ̸«̸
66@̸2(@̸K22̸@̸̸@̸̸K2̸3262%̸̸_Üĵ_qzz̶ʀR̸˶̸
̸H.̸
ŸH̸H]9̸]̸̸HE̸
d*r
using Raleigh’s
criterion, the smallest
separation
G3ì
+ ǵ Ôzǰ̸̸ɇƯÜ̸Sì
"
between two points that can be resolved is 1/(1.22Sì •fc)
". ǵ
½ÙÕ2à
¨ÙÕ2àµm
hr¡IJR̸rËÕ2àhr9\rnr ɟ¡ƣÿ̸
̸
QìK2̸2̸6̸&̸̸6̸2%i̸žg̸Ď62gƦ̸3g23gS̸ǃƗ̸
µØ¿à
dŠmin
=+0.146
zi˜˜̸›£M ı̸
66@̸2(@̸K22̸@̸̸@̸̸K2̸3262%̸̸_Üĵ_qzz̶ʀR̸˶̸
". ǵ
ǵ ,¹ǵ
!#. ǵ =i_˜Å̸Ŭ̸'3̸ Rì
Sì
zi˜˜̸›£M ı̸
!#. ǵ =i_˜Å̸ Ŭ̸'3̸ Rì
=i™̸ Ŭ̸%̸ {̸̸
_i˜̸čǤMe̸6̸g23̸6i̸
OTF
=i™̸ Ŭ̸%̸ {̸̸
_i˜̸čǤMe̸6̸g23̸6i̸
ĵ¼ǵ
63x, 1.4NA
ĵ¼ǵ
÷ǵ
4ǵ
,ºǵ
]¤˜ßϒìAì
äǵ
ê)ǵ
ƾǵ ¸£ƒÎĽÖ!ì
ñ)ǵ
ø»Äǵ
úǵ
(%#(
$"#$$# (&
$#( (#' (##(##( ( ((
; ì F x„ F J*Ò ?ì F x„ F J*Ò Aì F x„ F J*Ò ( Cì F x„ F J*Ò ( jöqǵ
=4¹ x„ 90Bìg0N.ì
Ƃ
!(
^¥˜ßϒìCì 5Ƃ*3tA*Ƃ733AƂ[[Ƃg³Ƃ‹Ƃ3A7]3Ƃ¡7ƂƂ45‹·AƂ7Ƃ
ö˜śƂ4¹Pì=Dì w„ j1mìg1N.ì[3AƂì=LìF‡äƂ x„ 91Bìg2N1ì3AƂìbÃSƂ‰ĐƂ˜Ƃ¬¤ƂAƂÄá‡ƂƂ
ă11ǵĚ·ǵJ¥»s҃¦ÈÒƪǥŦEǵµƒd¯©t°ÒǛEŷǵ Ú¸ðǵJ%ғt¢¶Òˆ·Ò ũ3ƂĝƂßRbƂ J%Ò ł3¬TƂ
ï® ̸̸]̸®̸Z]
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from arjournals.ann
by University of California - San Francisco on 05/09/10. For personal us
OTF, PSF of a defocused lens
¹©”ľ»Ì9Tà
'" 'X
%2%Q-Ò
Ùǵ
(ǵ
Pǵ
(ǵ
ǵ
¹©”ľ»Ì9Tà
F@ãì Ø|îǵ <$3$ǵ ;AHUǵ
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from arjournals.annualreviews.org
by University of California - San Francisco on 05/09/10. For personal use only.
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from arjournals.annualreviews.org
by University of California - San Francisco on 05/09/10. For personal use only.
ƹǵ
ęǵ
Ėǵ
25x lens 0 µm defocus
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from arjournals.annualreviews.org
by University of California - San Francisco on 05/09/10. For personal use only.
Annu. Rev. Biophys. Bioeng. 1984.13:191-219. Downloaded from arjournals.annualreviews.org
by University of California - San Francisco on 05/09/10. For personal use only.
>Eãì 6$Eì <}3~ǵ ;AHUǵ
ǵ
ƹǵ
4ǵ
ǵ
ǵ
ǵ
Pǵ
ãǵ
¶ǵ
¹ª”ž»Í:Œà
0ǵ
ƺǵ
ǵ
ǵ
ǵ
Pǵ
4ǵ
ãǵ
¶ǵ
¹ª”ž»Í:Œà
0ǵ
0ǵ
ïǵ
Ėǵ Ėǵ
Þà Þà
Kì Kì
ßà ßà
r„
r„
šì šì
& &
$& $&
ëì ëì
f„
ťǵ
ťǵ
ħǵ
25x lens 2.5 µm defocus
ħǵ
ïǵ
d’…®¡ƒ¹ùǵ #503:968*$(F+=4&;0549F,58F e҆æƂ x„ 5/Dì"
1*49FgÁÒB$8E04.F%5=4:9F5+F)*+5&>9FOì04F-5&?9FF8.4ì{Ƃ)*+5&@9F F†/bƂ{Ƃ)*+5&=9F`¹Dì{Ƃ
)*+5&A9
F!$'/F97@$8*F09F F5+F</*FD$C*2*4.:/F58FC46¹ ėµǵ
šì
&
$&
ëì
f„
ĺǵ
Nǵ
Ǫǵ
f„
25x lens 5 µm defocus
d’…®¡ƒ¹ùǵ #503:968*$(F+=4&;0549F,58F e҆æƂ x„ 5/Dì"
1*49FgÁÒB$8E04.F%5=4:9F5+F)*+5&>9FOì04F-5&?9FF8.4ì{Ƃ)*+5&@9F F†/bƂ{Ƃ)*+5&=9F`¹Dì{Ƃ
d’…®¡ƒ¹ùǵ #503:968*$(F+=4&;0549F,58F e҆æƂ x„ 5/Dì"
1*49FgÁÒB$8E04.F%5=4:9F5+F)*+5&>9FOì04F-5&?9FF8.4ì{Ƃ)*+5&@9F F†/bƂ{Ƃ)*+5&=9F`¹Dì{Ƃ
)*+5&A9
F!$'/F97@$8*F09F F5+F</*FD$C*2*4.:/F58FC46¹ ėµǵ
Þà
)*+5&A9
F!$'/F97@$8*F09F F5+F</*FD$C*2*4.:/F58FC46¹ ėµǵ
Kì
ħǵ
ǵ
d‘…¬¡¹ õǵ ĀFŁ`Ƃ Ęǵ FXƂ F `Ƃ Ƃ ?F`Ƃ F`=Ƃ =?Ƃ =Ƃ Ƃ XYƂ HHƂ [o„
pF²PƂ X`Ƃ =Ƃ X=?H
Ƃ 5Ƃ FƂ =Ƃ X=?HƂ xƂ x?HƂ PHƂ HƂ =Ƃ F?Ƃ ?§pĜƂ
¾üƂ ÿ
bûr
Ƃ
d‘…¬¡¹ õǵ ĀFŁ`Ƃ Ęǵ FXƂ F `Ƃ Ƃ ?F`Ƃ F`=Ƃ =?Ƃ =Ƃ Ƃ XYƂ HHƂ [o„
pF²PƂ X`Ƃ =Ƃ X=?H
Ƃ 5Ƃ FƂ =Ƃ X=?HƂ xƂ x?HƂ PHƂ HƂ =Ƃ F?Ƃ ?§pĜƂ
¾üƂ ÿ
bûr
Ƃ
ťǵ
ǵ
F@ãì Ø|îǵ <$3$ǵ ;AHUǵ
ǵ
0ǵ
(ǵ
Pǵ
ßà
r„
ƺǵ
Ùǵ
(ǵ
ęǵ
ĺǵ
ĺǵ Nǵ
Nǵ Ǫǵ
Ǫǵ
A
y
3D OTF
r
!
x
B
Fz
A
y
y
Only a bowl-shaped segment of the shell makes it through the objective: F
The intensity now becomes a convolution of the bowl with itself, which has
a a donut-shaped region of support. missing cone
xy
r
α
2x
!
x
C
B
Fz
0.6
0.4
y
0.2
Fxy
Fz
0
"0.2
"0.4
missing cone
"0.6
2x
1.5
1.5
1
0.5
1
0
C
Fy
"0.5
0.5
"1
"1.5
0
Figure 12.19.1 Calculation of widefield PSF frequency limits. (A) Cross-section of the 3D spherical cap that represents
the complex pupil function of the objective lens. The numerical aperture (NA) of the lens is defined by the refractive index
(RI ) of the immersion medium, and acceptance half angle (θ). The emission wavelength (λ) then determines the size of
the terms r, x, and y as calculated by Equation 12.19.1 and Equation 12.19.2. The observed spatial intensity function is
the square of the complex amplitude distribution, which corresponds to the auto-correlation of the spherical cap in the
frequency domain. The resulting 3D optical transfer function (OTF) is a 3D toroid with cross-section shown in (B). The
maximum lateral and axial frequencies are shown, which determine the resolution of the lens. (C) Cross-section of toroid
that encompasses the 3D OTF of a widefield microscope.
0.6
0.4
0.2
Fz
Fx
0
"0.2
"0.4
12.19.3
"0.6
Current Protocols in Cytometry
1.5
1.5
1
0.5
1
0
Fy
"0.5
0.5
"1
"1.5
Fx
0
Figure 12.19.1 Calculation of widefield PSF frequency limits. (A) Cross-section of the 3D spherical cap that represents
the complex pupil function of the objective lens. The numerical aperture (NA) of the lens is defined by the refractive index
(RI ) of the immersion medium, and acceptance half angle (θ). The emission wavelength (λ) then determines the size of
the terms r, x, and y as calculated by Equation 12.19.1 and Equation 12.19.2. The observed spatial intensity function is
the square of the complex amplitude distribution, which corresponds to the auto-correlation of the spherical cap in the
frequency domain. The resulting 3D optical transfer function (OTF) is a 3D toroid with cross-section shown in (B). The
maximum lateral and axial frequencies are shown, which determine the resolution of the lens. (C) Cross-section of toroid
that encompasses the 3D OTF of a widefield microscope.
12.19.3
Current Protocols in Cytometry
Supplement 52
Supplement 52
Experimentally measured OTF
z
r
z
• Intensity is very peaked at origin
• data along Z direction is missing
r
Experimentally measured PSF
z
x
PSF, OTF & deconvolution
In real space:
=
Observed Image(r)
True Object(r)
Point Spread
Function, PSF(r)
In reciprocal space the convolution becomes a product:
PSF is called the
“Optical Transfer Function”,
OTF
Image = Object • PSF
This suggests:
Object
=
Image
OTF
???
(“Deconvolution”)
PSF, OTF & deconvolution
In real space:
=
Observed Image(r)
True Object(r)
Point Spread
Function, PSF(r)
In reciprocal space the convolution becomes a product:
PSF is called the
“Optical Transfer Function”,
OTF
Image = Object • PSF
This suggests:
Object
=
Image
OTF
???
(“Deconvolution”)
Whatʼs the catch??
A: We canʼt divide by OTF(k) if it is zero (or small because of noise)
Deconvolution strategies
Nearest neighbor: simplest method, only takes into account adjacent sections
ie subtract out blurred version of adjacent sections from central section
Ij
≈ c1[ Oj - c2(PSFΔz ⊗ Oj+1 + PSF-Δz ⊗ Oj-1) ]
for speed do the convolutions as Fast Fourier Transforms (FFT), multiplication, FFT-1
PSFΔz ⊗ Oj+1 = FFT-1[ FFT(Oj+1) • OTFΔz) ]
optical section from a DAPI
stained polytene nucleus
before and after nearest neighbor
Deconvolution strategies
much better to consider the contributions of all the sections to one another = 3D
Weiner filter: simplest 3D method,
Object
=
linear processing, takes care of zeros
Image
OTF + γ
γ is related to the signal to noise, sets maximum amplification
or if OTF is complex:
Image • OTF*
OTF•OTF* + γ
Deconvolution strategies
even better to include a priori knowledge about solution such as positivity: Object ≥ 0
family of iterative constrained methods
work by calculating convolution, followed by update
image first guess
new
estimate
blur with
PSF
constraints
calculate
correction
calculate
error
PSF
reblurred
observed image
Figureupdate
12.19.3 (vanCittert’s
Block diagram ofmethod)
the iterative deconvolution process. The image estimate is
arithmetic
blurred with the PSF to form the reblurred image, which is then compared with the observed
start image.
withThe
I0 image
= Odifferences and any other constraints are used to form a correction update that
creates an improved image estimate. The process is repeated for a number of iterations until a
suitable result is achieved.
Ik+1 = Ik + (O - PSF ⊗ Ik)
ˆ
with positivity constraint: if Ik+1 <0 then set Ik+1 = 0
ˆ

g
⋅ h ∗
f k +1 = f kmethod)
or use multiplicative update (Gold’s
 h ⊗ fˆ
k
Ik+1 = Ik •
O
PSF ⊗ Ik



Equation 12.19.11
where ⊗ is the convolution operation, and * is the correlation operation. This ML-EM
algorithm is often referred to as Richardson-Lucy (RL) iterations (Richardson, 1972;
Lucy, 1974) and has several useful characteristics, including guaranteed pixel positivity,
stable convergence, and the ability to recover frequency components outside the OTF
(which can reduce ringing artifacts that occur at feature edges). The initial estimate used
can be just the observed data itself, or the result of another filtering operation, such as the
Wiener filter. The closer the estimate is to the solution, the fewer number of iterations
required. The standard RL iterations can be slow to converge, so acceleration techniques
are normally used to reduce the number of iterations and the amount of processing
required. The convolution and correlation operations require a total of four 3D Fourier
transform operations per iteration (assuming the PSF is precalculated), which is the bulk
of the computations performed.
Another iterative algorithm that is often implemented is Gold’s method (Gold, 1964;
Sibarita, 2005), which has a similar form to the RL equation; however, it is not a
maximum likelihood technique:
 g 
fˆk +1 = fˆk ⋅ 

 h ⊗ fˆk 
Equation 12.19.12
Each iteration requires less computation; however, Gold’s method tends to amplify noise,
which requires a smoothing operation to be applied at regular intervals (e.g., using a 3D
Gaussian filter every three iterations), in order to retain stability.
BLIND DECONVOLUTION
3D Deconvolution
Microscopy
12.19.10
Supplement 52
Another method of determining the microscope PSF is to estimate it directly from the
observed data itself. The ability to separate the PSF and underlying object from a single
observation may, at first, seem counter-intuitive; however, physical constraints such as
Current Protocols in Cytometry
One approach to implementing blind deconvolution uses alternating cycles of the RL
algorithm described above. Initially, the PSF is fixed and the image estimate is updated,
and then the image is fixed and the PSF estimate updated. This process is repeated,
with each cycle making incremental changes to both estimates until they converge to a
solution, as shown in Figure 12.19.4. A simple implementation for estimating both the
image and PSF (Fish et al., 1995) is:
Deconvolution strategies

fˆ = fˆ ⋅ h ∗ 
what if your PSF is not accurate?
h
k +1
k
k
k

hˆk +1 = hˆ k ⋅ fˆk +1 ∗ 
g 

⊗ fˆk 
g


⊗ fˆ Image and PSF
blind deconvolution seeks to estimate hboth
k
k +1
Equation 12.19.13
Image update
PSF update
image
new image
estimate
blur with
PSF
PSF
new PSF
estimate
blur with
image
calculate
correction
calculate
error
PSF
constraints
calculate
correction
calculate
error
image
constraints
observed
image
Figure 12.19.4 Block diagram of the iterative blind deconvolution process using alternating update cycles. First, the PSF
is fixed and the image is updated, then the image is fixed and the PSF updated. With appropriate starting estimates,
constraints, and a priori knowledge, a suitable solution for the image can be found while the PSF is adapted to better fit
the data.
Cellular and
Molecular
Imaging
12.19.11
Current Protocols in Cytometry
Supplement 52
Some comparisons: Hela cells, DAPI stained
A
B FITC (528 nm)
DAPI (457 nm)
C Red (617 nm)
Texas
D
Cy-5
(685(457
nm)nm)
DAPI
E
FITC (528 nm)
Texas Red (617 nm)
Cy-5 (685 nm)
Y
Z
BX
original
nearest neighbor
wiener filter
Gold’s method
blind
F
E following page) Maximum
C
Figure 12.19.5 (continued on
intensity XY and XZ projections of four
channels from a HeLa cell undergoing mitosis with results from a variety of deblurring and deconvolutions algorithms. (A) Original widefield data, (B) Nearest Neighbors at 95% setting, (C) Wiener
filtering, (D) Gold’s method with 10 iterations (smoothing every 3 iterations), (E) iterative MLE (10
now many related variationsaccelerated iterations), and (F) blind iterative MLE (10 accelerated iterations). Each volume is 640
× 640 pixels with 50 nm pixels and 95 Z-slices spaced 200 nm apart. XZ projections are stretched
axially by a factor of 4 to give cubic voxels. Original data is courtesy of Jason Swedlow, University
practical issues are signal toof noise,
Dundee.accuracy of PSF
Reading
a variety
of image file
formats, including native files from the acquisition
if sample is very thick then OSF
can vary
throughout
sample
Reading image meta-data (e.g., pixel spacings, objective parameters, wavelengths)
(minimize by matching index of
refraction
Displaying optical slices and 3D projections
C
Handling multichannel and time-series volumes
Cellular and
Preprocessing the data
for illumination
non-uniformities
or pixel errors
F to correct
Molecular
Figure
12.19.5
(continued)
Figure 12.19.5 (continued on following page) Maximum
intensity XY
and XZ
projections
of four
Imaging
Setting upmitosis
the parameters
theaalgorithm
channels from a HeLa cell undergoing
with resultsfor
from
variety of deblurring and deconvolutions algorithms. (A) Original widefield data, (B) Nearest Neighbors at 95% setting, (C) Wiener
12.19.13
Algorithm memory requirements
filtering, (D) Gold’s method with 10 iterations (smoothing every 3 iterations), (E) iterative MLE (10
Limits
on
size
of
volume
that
can
be
processed
accelerated iterations), Current
and (F)Protocols
blind iterative
MLE (10 accelerated iterations). Each volume is 640
in Cytometry
Supplement 52
Effective are
usage
of the CPU and multiple processors
stretched
× 640 pixels with 50 nm pixels and 95 Z-slices spaced 200 nm apart. XZ projections
axially by a factor of 4 to give cubic voxels. Original data is courtesy of Jason
Swedlow,
University
Total
algorithm
processing time
of Dundee.
Automated batch processing of multiple datasets
Visualization and comparison of results with the original data
Reading a variety of image file formats, including native files Output
from theofacquisition
data in format suitable for further analysis
Reading image meta-data (e.g., pixel spacings, objective parameters, wavelengths)
Since deconvolution is an integral part of the imaging process, the workflow from data
Displaying optical slices and 3D projections
acquisition to processed results should be as seamless as possible to reduce the amount
Handling multichannel and time-series volumes
Cellular
and
of
manualordata
handling,
employing
image restoration becomes routine.
Figure 12.19.5 (continued on
following page)
intensity
and
XZ
projections
of
four
3D
Deconvolution
Preprocessing
theMaximum
data to correct
forXYillumination
non-uniformities
pixel
errors such that
Molecular
Figure 12.19.5 (continued)
Microscopy
Imaging
channels from a HeLa cell undergoing
mitosis
with
results
from
a
variety
of
deblurring
and
deconSetting up the parameters for the algorithm
volutions algorithms. (A) Original widefield data, (B) Nearest Neighbors at 95% setting, (C) Wiener
12.19.14
12.19.13
Algorithm
memory
filtering, (D) Gold’s method with 10 iterations (smoothing every 3 iterations),
(E) iterative
MLE (10requirements
accelerated iterations), and (F) blind iterative MLE (10 accelerated iterations).
Eachon
volume
is 640
Limits
size
of
volume
that
can
be
processed
Supplement 52
in-slices
Cytometry
Supplement 52
spaced 200 nm apart. XZ projections
stretched
× 640 pixels with 50 nmCurrent
pixelsProtocols
and 95 Z
Effectiveare
usage
of the CPU and multiple processors
axially by a factor of 4 to give cubic voxels. Original data is courtesy of Jason Swedlow, University
Total algorithm processing time
of Dundee.
Current Protocols in Cytometry
Automated batch processing of multiple datasets
Visualization and comparison of results with the original data
Reading a variety of image file formats, including native files from the acquisition
Output of data in format suitable for further analysis
Reading image meta-data (e.g., pixel spacings, objective parameters, wavelengths)
Displaying optical slices and 3D projections
Since deconvolution is an integral part of the imaging process, the workflow from data
Handling multichannel and time-series volumes
acquisition to processed resultsCellular
shouldand
be as seamless as possible to reduce the amount
Preprocessing the data to correct for illumination non-uniformities
ordata
pixelhandling,
errors such Molecular
of manual
that employing image restoration becomes routine.
3D Deconvolution
Imaging
Setting up the parameters for the algorithm Microscopy
12.19.14
Current Protocols in Cytometry
Supplement 52
12.19.13
Supplement 52
Current Protocols in Cytometry
sample η =
mounting media η
sample η ≠
mounting media η
This causes
Spherical Aberration
sample η not constant
Depth-Dependent Effects in Water:
Simulations of PSFs
Hanser
Depth Dependent Deconvolution: Biological Data
Line Profile Through 1 Spot in Nucleus
Section ~7 .6 µm into sample
Line Profile:
Measured Data
Post-Deconvolution
Spatially-Invariant
Spot FWHM: "
"
"
"
"
Measured
Spatially Invar.
Depth-Dep.
Depth-Dependent
Measured = 0.28 µm
Spatially-Invariant = 0.20 µm
Depth-Dependent = 0.11 µm
Hanser
Adaptive optics can correct for depth dependent effects
(reshape optical wavefront)
ADAPTIVE OPTICS MICROSCOPY
139
Fig. 2. Microscope layout. Grey represents the emission path. The blue striped beam is the excitation light and the red striped beam is the reference beam.
See text for details.
Three-dimensional image stacks
All the three-dimensional data stacks were taken using the
piezo for focusing through the sample except the data for
Figs 10 and 11 in which the focusing was performed by the DM.
In three-dimensional image stacks in which Eq. (1) was used
to correct the depth aberration, the correction was adjusted for
each depth so that the PSF is corrected throughout the sample
and not just at one particular depth. The only exception to
this is the images of a bead in Figs 4 and 6, where the phase
correction was set to the depth of the centre of the bead and
not adjusted for the three-dimensional data stack.
Deformable mirror control
For the DM, we chose the Mirao52D from Imagine-Optic
(www.imagine-optic.com) because it is capable of large
displacements and thus permits the correction of aberrations
deep into a sample. The 15-mm-diameter mirror has 52
actuators on a square grid with 2.5 mm spacing. The mirror is
capable of a maximum displacement of ±75 µm for the focus
mode (Z 02 ) and ±8 µm for the first order spherical aberration
(Z 04 ). The mirror can take the shape of any Zernike mode
through order 4 with a root-mean square (rms) wavefront
error of less than 20 nm. Because the mirror cannot set the
higher order terms, the Strehl ratio will degrade with depth,
as the amount of higher order spherical terms needed to fit
24
the correction (given by Eq. 2) increases. Thus, the maximum
depth the mirror can correct will be limited by the residual
aberrations and not the maximum displacement of the focus
mode.
We control the mirror by measuring the wavefront of the
HeNe laser with the wavefront sensor (reference path in
Fig. 2). We reference the wavefront to a measurement with
all actuators set to zero, and then measure the wavefront
on a 32 × 32 lenslet array for each of the 52 actuators
activated individually, yielding a 1024 × 52 matrix. To set
a desired mirror shape, we use the standard singular value
decomposition technique (Gavel, 2003) to determine the
matrix S which will yield the actuator values for a desired
wavefront. Typically, we only retain the first 45 singular
values.
Sample preparation
To estimate the PSF, we imaged 200-nm-diameter YellowGreen fluorescent beads (F-8811, Molecular Probes, Inc.,
"
C 2009 The Authors
C 2009 The Royal Microscopical Society, Journal of Microscopy, 237, 136–147
Journal compilation "
Adaptive optics can correct for depth dependent effects
142
flat mirror
P. KNER ET AL.
mirror shaped to correct
spherical aberration
144
P. KNER ET AL.
raw
decon
X-Y
-AO
Fig. 5. Back pupil plane calculated by phase retrieval from the threedimensional image of a 200 nm bead at the cover slip. Amplitude (left)
and phase scaled from –π/2 to π/2 radians (right). The DM print through
is clearly visible. 20 of the actuators are at the edge of the mirror and
therefore do not produce a print-through bump.
achieves several important results. First, the peak intensity
of the corrected images is a factor of two larger than the
uncorrected image. Because the uncorrected image was taken
first, any bleaching of the sample (measured from similar data
to be 4% per image stack) would result in a higher intensity
for the uncorrected image, underestimating the improvement
obtained using the DM. Second, the correction removes the
low intensity ‘pedestal’ of light around the main peak in the
uncorrected image and returns it to the central peak as can
be clearly seen in the logarithmic scale image, Fig. 6(c) and
(d). Third, the shape of the PSF (compare Fig. 6e and f) after
correction is almost exactly the same as that of the PSF at the
cover slip, which should result in much better deconvolution
results. Lastly, the width of the peak in the axial direction
is significantly reduced after correction resulting in a higher
axial resolution (Fig. 6h). The lateral plane FWHM of the
peak is not significantly changed by the correction as seen in
Fig. 6(g). This is because the depth aberrations do not increase
the width of the central peak very much. The aberrations
create a broad pedestal upon which the central peak sits (Fig.
6c and d). Figure 6(i) and (j) show simulations of the PSF
in the axial direction for parameters corresponding to the
measured data. Figure 6(i) simulates a PSF 65 µm below the
cover slip with the same refractive index mismatch and Fig.
6(j) simulates a well-corrected PSF at the cover slip. As can
be seen in the figure, the agreement is quite good. The main
difference is that in the experiment the correction increases
the peak intensity by 2.05. In theory, the peak intensity of the
corrected PSF is 3.75 times higher than the aberrated PSF.
Biological imaging
Of course, the goal of using adaptive optics in fluorescence
microscopy is to correct images of biological samples. In this
section we will show results demonstrating the correction of
spherical aberration in biological samples, but the results are
less dramatic than the results on a single bead because the
biological sample introduces scattering and other aberrations.
Figure 7 shows uncorrected (Fig. 7a) and corrected (Fig. 7b)
X-Y log
scale
+AO
X-Z
Fig. 9. Deconvolved images of alexa488-phalloidin labelled B16F10 mouse cells. Images are 4.4 µm below the cover slip. (a) Uncorrected image. (b)
Uncorrected deconvolved image (c) image corrected by adaptive optics. The correction assumed a refractive index of 1.38 (d) image corrected by adaptive
optics after deconvolution. The scale bar is 5 µm.
Alexa 488-phalloidin stained mouse cells:
images at 4.4µm below cover slip
due to depth correction may be negated by aberrations in
optical components that are uncorrected as the system is
focused into the sample. Figure 11 is an example of using
a DM to focus through an adult C. elegans worm expressing
a GFP sur-5 construct. The images in Fig. 11(a) are from a
three-dimensional data stack taken using the DM to focus in
z, and are comparable to the images taken in Fig. 11(b) using
mechanical focusing. The same features are clearly visible in
both images although the level of background fluorescence has
changed.
Fig. 6. Images of a 200 nm bead 67 µm below the cover slip in a
water/glycerol mixture with n = 1.42. (a) Uncorrected image of in-focus
plane. (b) Corrected image of in-focus plane: same scale as (a). (c) and (d)
are the same as (a) and (b), respectively, but on a logarithmic scale. (e)
and (f) are cross-sections through the focal plane on a linear scale. The
scale bars are 1 µm. (g) and (h) are line profiles through of the intensity
through the centre of the bead along a lateral and the longitudinal axis,
respectively. The dashed line is from the uncorrected image and the solid
line is from the corrected image. (i) and (j) are simulations of the PSF.
(i) corresponds to the uncorrected PSF 65 µm into a material with index
1.42 using a 1.2NA objective with a 1.512 refractive index immersion
oil. (j) is a simulated PSF at the cover slip. The peak intensity for (j) is
3.75 times the peak intensity for (i).
lateral images 24 µm below the cover slip of UMUC bladder
cancer cells with GFP-TRF1 labelled telomeres. Although the
intensity of the cell autofluorescence is not increased by the
correction of the depth aberrations, the GFP-TRF1 signal is
!
C 2009 The Authors
C 2009 The Royal Microscopical Society, Journal of Microscopy, 237, 136–147
Journal compilation !
Fig. 10. Maximum intensity projections of three-dimensional data stacks of 200 nm beads in agarose. The vertical direction is the axial direction and
the total height of each image represents 22.5 µm. The width of each image is 34 µm. The arrows point to the position of the cover slip. Image (a) was
taken using the piezo for focusing. Image (b) was taken using the DM to focus. The DM shape was set by Eq. (2) using a sample refractive index of 1.34.
(c) Shows close-ups, (i) and (ii), of the bead indicated by the red arrows in (a) and (b). The images are 1.6 µm × 8 µm (16 × 16 pixels) cross-sections in
the xz-plane, scaled to maximum intensity.
"
C 2009 The Authors
C 2009 The Royal Microscopical Society, Journal of Microscopy, 237, 136–147
Journal compilation "
Structured illumination microscopy:
the idea
Two patterns
superposed
multiplicatively
give rise to
moiré fringes
The moiré fringes
may be coarse enough
to resolve
even if neither
original pattern is
• Illuminate the sample with a light pattern
• Observe moiré fringes between the pattern and the sample structure
• Deduce otherwise unresolvable information about the sample
Illuminate sample with parallel stripes
Observable
region
Observable
region
The emitted light contains 3 superimposed
information components shifted by 0, ± the inverse stripe spacing
Record 3 images (0°, 120°, 240° shifts) to sort out 3 components
Resolution extension by
Structured Illumination
ky
Normally
observable
information
ky
kx
kx
Spatial frequency of
illumination pattern
…and can repeat in many directions
Information
observable as
moiré fringes
illumination intensity in real space
z
x
Observable Region
Conventional
Structured
Illumination
Microscope
(2
(3
(1orientations)
orientation)
What are the limits of
Structured Illumination?
Normally
observable
information
Information
observable as
moiré fringes
ky
kx
Spatial frequency of
illumination pattern
• Linear theory says this is impossible
• How about exploiting non-linear processes?
A simple source of nonlinearity: saturation
One photon
per lifetime
Emission intensity
(R. Heintzmann
Max-Planck Gottingen)
1
0.8
0.6
0.4
0.2
1
2
3
4
Illumination intensity
One photon
per absorption cross section
per lifetime
5
Saturated Structured Illumination
FT (log scale)
5
4
nsity
3
tion inte
2
0.5
Illumina
0.75
ion
Emiss
intens
0
1
0.25
0
0
2
X
ity
1
4
6
Resolution extension
by nonlinear structured illumination
Effective observable regions
Conventional
microscopy
Linear
Structured
illumination
3 directions
200 nm res.
100 nm res.
Nonlinear structured illumination
2 new harmonics, 8 directions
50 nm res.
Non-linear structured illumination
Linear
structured
illumination
Conventional
microscopy
1 µm
~250 nm resolution
(diffraction limit)
1 µm
~120 nm resolution
50 nm microspheres
Saturated
structured
illumination
1 µm
46 nm resolution!
Drosophila embryo section DNA/RNA stain
Conventional
microscopy
Min. FWHM ≈ 280 nm
Linear
structured
illumination
Min. FWHM ≈110 nm
Saturated
structured
illumination
(1 new harmonic)
Min. FWHM ≈ 80 nm
Going beyond the diffraction limit:
more light collection angles
We know:
Higher NA
Gathering light over
larger set of angles
Higher resolution
…So what about gathering the light
emitted toward the back side?
OTF when detecting through two lenses
Through two
objectives
Through one
objective
Free space
ky
ky
ky
support of
α
kz
A(k)
kz
kz
Convolved
with itself:
Convolved
with itself:
ky
Convolved
with itself:
ky
ky
support of
OTFdet(k)
kz
kz
kz
I5M concept
I5M OTF
CCD
Incoherent
light
source
Detection
angles
Illum.
angles
OTFdet
illum
OTFeff =
=
OTFdet
illum
OTFeff
Combine methods: I5S
use two lenses to collect more angles
Combination of :
1. Imaging Interference – detection
through 2 opposing objectives
2. Structured Illumination
Observable Region through 2 opposing objectives
Structured
Uniform
Illumination
illumination
(1 orientation)
Structured
Illumination
(3 orientations)
– I5S
Comparison of Resolving Power
Conventional
Structured Illum.
I 5S
y
x
0.136µm
z
0.122µm
x
Sample: 0.12µm red-fluorescence microshperes
Lin Shao, Mats Gustafsson
Drosophila Anaphase Chromosomes (.2µm wide)
Lin Proc.
Decon
Decon + LCE
Lin Shao
Comparison between OM and EM-Tomo
EM
OM