Light field microscopy
Marc Levoy, Ren Ng, Andrew Adams
Matthew Footer, Mark Horowitz
Stanford Computer
Graphics Laboratory
Executive summary
• captures the 4D light field inside a microscope
• yields perspective flyarounds and focal stacks from a single
snapshot, but at lower spatial resolution
• focal stack → deconvolution microscopy → volume data
Marc Levoy
Devices for recording light fields
(using geometrical optics)
big
scenes
small
scenes
•
•
•
•
•
handheld camera
camera gantry
array of cameras
plenoptic camera
light field microscope
[Buehler 2001]
[Stanford 2002]
[Wilburn 2005]
[Ng 2005]
(this paper)
Marc Levoy
Light fields at micron scales
• wave optics must be considered
– diffraction limits the spatial × angular resolution
• most objects are no longer opaque
– each pixel is a line integral through the object
» of attenuation
» or emission
– can reconstruct 3D structure from these integrals
» tomography
» 3D deconvolution
Marc Levoy
Conventional versus plenoptic camera
Marc Levoy
Conventional versus plenoptic camera
125μ
square-sided microlenses
uv-plane
st-plane
Marc Levoy
Digital refocusing
Σ
Σ
• refocusing = summing windows
extracted from several microlenses
Marc Levoy
Example of digital refocusing
Marc Levoy
Refocusing portraits
Marc Levoy
Macrophotography
Marc Levoy
Digitally moving the observer
Σ
Σ
• moving the observer = moving the
window we extract from the microlenses
Marc Levoy
Example of moving the observer
Marc Levoy
Moving backward and forward
Marc Levoy
A light field microscope (LFM)
eyepiece
intermediate
image plane
objective
specimen
Marc Levoy
A light field microscope (LFM)
sensor
eyepiece
intermediate
image plane
objective
specimen
→ reduced lateral resolution on specimen
= 0.26μ × 12 spots = 3.1μ
• 40x / 0.95NA objective
↓
0.26μ spot on specimen
× 40x = 10.4μ on sensor
↓
2400 spots over 25mm field
• 1252-micron microlenses
↓
200 × 200 microlenses with
12 × 12 spots per microlens
Marc Levoy
A light field microscope (LFM)
sensor
2.5mm
160mm
0.2mm
Marc Levoy
Example light field micrograph
•
•
•
•
•
•
orange fluorescent crayon
mercury-arc source + blue dichroic filter
16x / 0.5NA (dry) objective
f/20 microlens array
65mm f/2.8 macro lens at 1:1
Canon 20D digital camera
200μ
ordinary microscope
light field microscope
Marc Levoy
The geometry of the light field
in a microscope
• microscopes make orthographic views
• translating the stage in X or Y
provides no parallax on the specimen
f
• out-of-plane features don’t shift
position when they come into focus
objective lenses
are telecentric
Marc Levoy
Panning and focusing
panning sequence
focal stack
Marc Levoy
Mouse embryo lung
(16x / 0.5NA water immersion)
200μ
light field
pan
focal stack
Marc Levoy
Axial resolution
(a.k.a. depth of field)
• wave term + geometrical optics term
DOFtot 
λn
n

e
2
NA
M  NA
• ordinary microscope (16x/0.4NA (dry), e = 0)

0.535  1
 3.3
2
0.4
(wave optics dominates)
• with microlens array (e = 125μ)

(geometrical optics dominates)
0.535  1
1

 125  3.3  19.5  22.8
2
0.4
16  0.4
• stopped down to one pixel per microlens
 3.3  19.5  12 spots  237
→ number of slices
in focal stack
= 12
Marc Levoy
3D reconstruction
• confocal scanning
[Minsky 1957]
• shape-from-focus
[Nayar 1990]
• deconvolution microscopy
[Agard 1984]
– 4D light field → digital refocusing →
3D focal stack → deconvolution microscopy →
3D volume data
(UMIC SUNY/Stonybrook)
(Noguchi)
(DeltaVision)
Marc Levoy
3D deconvolution
[McNally 1999]
focus stack of a point in 3-space is the 3D PSF of that imaging system
•
•
•
•
•
•
object * PSF → focus stack
 {object} ×  {PSF} →  {focus stack}
 {focus stack}   {PSF} →  {object}
spectrum contains zeros, due to missing rays
imaging noise is amplified by division by ~zeros
reduce by regularization, e.g. smoothing
 {PSF}
Marc Levoy
Silkworm mouth
(40x / 1.3NA oil immersion)
100μ
slice of focal stack
slice of volume
volume rendering
Marc Levoy
Insect legs
(16x / 0.4NA dry)
200μ
volume rendering
all-focus image
[Agarwala 2004]
Marc Levoy
3D reconstruction (revisited)
• 4D light field → digital refocusing →
3D focal stack → deconvolution microscopy →
3D volume data
(DeltaVision)
• 4D light field → tomographic reconstruction →
3D volume data
(from Kak & Slaney)
Marc Levoy
Implications of this equivalence
• light fields of minimally scattering volumes
contain only 3D worth of information, not 4D
• the extra dimension serves to reduce noise,
but could be re-purposed?
Optical
Projection
Tomography
[Sharpe 2002]
Marc Levoy
Conclusions
• captures 3D structure of microscopic
objects in a single snapshot, and at a
single instant in time
Calcium fluorescent imaging
of zebrafish larvae optic tectum
during changing visual stimula
Marc Levoy
Conclusions
• captures 3D structure of microscopic
objects in a single snapshot, and at a
single instant in time
but...
• sacrifices spatial resolution to obtain
control over viewpoint and focus
• 3D reconstruction fails if specimen is
too thick or too opaque
Marc Levoy
Future work
• extending the field of view by correcting
digitally for objective aberrations
Nikon 40x 0.95NA (dry) Plan-Apo
Marc Levoy
Future work
• extending the field of view by correcting
digitally for objective aberrations
• microlenses in the illumination path
→ an imaging microscope scatterometer
200μ
angular dependence
of reflection from
single squid iridophore
Marc Levoy
http://graphics.stanford.edu/projects/lfmicroscope
Marc Levoy