Pupil Phase Encoding for Multi-Aperture Imaging
V. P. Paucaa , D. Chen c , J. van der Grachtd R. J. Plemmonsa,c , S. Prasadg , T. C. Torgersena ,
a
g
Department of Computer Science, Wake Forest University, Winston-Salem, NC 27109
c Department of Mathematics, Wake Forest University, Winston-Salem, NC 27109
d Holospex, Inc., 6470 Freetown Rd., Suite 200-104, Columbia, MD 21044
Center for Advanced Studies and Department of Physics and Astronomy, University of New
Mexico, Albuquerque, NM 87131
ABSTRACT
Digital super-resolution refers to computational techniques that exploit the generalized sampling theorem to
extend image resolution beyond the pixel spacing of the detector, but not beyond the optical limit (Nyquist
spatial frequency) of the lens. The approach to digital super-resolution taken by the PERIODIC multi-lenslet
camera project is to solve a forward model which describes the effects of sub-pixel shifts, optical blur, and
detector sampling as a product of matrix factors. The associated system matrix is often ill-conditioned, and
convergence of iterative methods to solve for the high-resolution image may be slow.
We investigate the use of pupil phase encoding in a multi-lenslet camera system as a means to physically
precondition and regularize the computational super-resolution problem. This is an integrated optical-digital
approach that has been previously demonstrated with cubic type and pseudo-random phase elements. Traditional
multi-frame phase diversity for imaging through atmospheric turbulence uses a known smooth phase perturbation
to help recover a time series of point spread functions corresponding to random phase errors. In the context of a
multi-lenslet camera system, a known pseudo-random or cubic phase error may be used to help recover an array
of unknown point spread functions corresponding to manufacturing and focus variations among the lenslets.
1. INTRODUCTION
Digital super-resolution refers to the technique of extending the resolution of an image beyond that allowed by
the detector pixel size or pixel pitch (but still below the limit provided by the lens aperture). This technique
is most often related to the use of a set of low-resolution images of a subject obtained with a single lens over
time or obtained through a multi-lens imaging system in a single exposure. The latter approach has recently
been explored by a number of research efforts and notably by the thin observing module with bounded optics
(TOMBO),7 MONTAGE,5 and PERIODIC9 projects. These multi-lenslet camera systems enable the use of many
types of observational diversities, including phase, polarization, neutral density, and wavelength diversities.
Figure 1 illustrates an example 3 × 3 PERIODIC camera system. A lenslet array forms multiple images of a
subject on a single detector. Under nearly all observing conditions, the multiple images focused on the detector
will be sampled with slightly different alignment of the image with respect to the detector pixels. The ensemble
of sub-pixel shifted low-resolution images encodes subject information at higher spatial frequencies than present
in each image individually.10 Numerical methods for digital super-resolution reconstruct a single image from
the ensemble, combining the available high frequency information to produce a high-resolution or super-resolved
image.
Recent PERIODIC imager prototypes have been designed to allow post-processing multiple observations made
with differing neutral density, IR, polarization, and cubic phase filters, for multi-modality imaging.2, 3 Figure 2(a)
shows a picture of the array imaging component. The lenslet array consists of 18 2mm sub-apertures containing
eight clear lenses, three lenses at different focal planes, three neutral density, two polarized, and two spectral
(650nm and near-IR) filters that are able to capture an information-rich image over a single 10 megabit detector
plane. The pixel pitch is 7.9µm. The digital component (see Figure 2(b)) consists of a Matlab-based graphical
user interface front-end and a computational back-end. The front-end follows a flexible object-oriented design
that is self-reconfigurable with respect to the camera lenslet and detector specifications. The back-end consists of
a set of numerical methods for efficiently solving sub-image registration, digital super-resolution reconstruction,
Figure 1. A visual overview of a PERIODIC imaging system.
(a) PERIODIC array imaging prototype.
(b) PERIODIC digital component front-end.
Figure 2. Hardware and software components of a PERIODIC imager.
and deconvolution problems. This GUI enables realtime imaging through any user-selected lenslet and near-real
time super-resolution and dynamic range reconstruction using an appropriately selected subset of lenslets.
In spite of these advancements in observational diversity, the full computational imaging system design space
has not been fully explored. Recently, Ashok and Neifeld1 described a way to exploit the pupil phase to obtain
additional diversity from a set of subpixel shifted measurements, in a process called PSF engineering. In this
paper, we explore PSF engineering through the use of cubic phase masks to physically precondition the underlying
computational problem, leading to improved super-resolution performance. Cubic and other types of polynomial
phase masks have been traditionally used to extend a single channel imaging system’s depth of field, without
incurring on a loss of light influx, see e.g. Prasad et al.11 and references therein. The present work is based on
the observation that polynomial phase masks can also increase the restorability of an out-of-focus measurement
by boosting the singular values of the system matrix above the noise floor.8 This process is akin to numerical
preconditioning methods employed in the solution of many imaging and other inverse problems, and constitutes
a form of physical preconditioning (through optical lenses) that takes place before the data is sampled. From a
practical standpoint, a cubic phase mask also helps mitigate the effects of aberrations and focal length variations
due to manufacturing errors, from one lenslet to the next.11
The remainder of this paper is structured as follows. We introduce the forward image formation model for a
multi-lens imager and the associated digital superresolution inverse problem in Section 2. The approach of using
cubic phase masks in the design of extended optical PSFs to improve the resolution performance of multi-lens
imaging systems is outlined in Section 3. Resolution performance results from this approach on simulation data
are given in Section 4, followed by conclusions and final remarks.
2. IMAGING SYSTEM MODEL
2.1. The Image Formation Model
We consider a multi-lens imaging system consisting of a maximum of ` sub-apertures of similar optical characteristics. The continuous signal formed in the image plane, with coordinate x = (x1 , x2 ), through each of these
sub-apertures is given by
Z
ĝj (x) =
fc (u)h(x − u + j4u)du,
(1)
where fc represents the continuous object, j4u is the subpixel shift due to the relative position of the jth subaperture, and h is the optical system’s point spread function (PSF). We model the PSF in the image plane for
the jth lenslet as
2
Z
1
2π T
x r
ıφj (r)+ı λf
(2)
dr ,
P (r)e
hj (x) = √
λf A
where r = (r1 , r2 ) is a coordinate in the two-dimensional pupil plane, P (r) is the pupil function equal to 1 inside
the pupil and 0 outside for a clear pupil, φj (r) is the pupil phase, λ is the wavelength of the illuminating light,
f is the focal length, and A is the area of the pupil. For a diffraction-limited imaging system we have φj (r) = 0.
If the lenslet includes a cubic phase mask then,
φj (r) = α(r13 + r23 ),
(3)
where α is a parameter (in units of λ) that denotes the strength of the phase mask. Typical cubic phase masks
for depth of field extension have α > 20.
A detector discretizes ĝj with a sampling rate inversely proportional to the the pixel pitch. We use a rectangle
function as the sampling pixel function, resulting in a discrete signal given by
0
Z
x − iT
0
gj (i) = ĝj (x ) rect
dx0 + η(i),
(4)
T
where i denotes the ith pixel in the detector, T is the pixel pitch, and η is additive measurement noise. For the
purpose of this paper, we neglect nonlinearities inherent in the quantization of amplitude into a finite number of
discrete intensity values, determined by the number of bits available on each pixel. If ĝj is a bandlimited signal
(|ω| ≤ B), where B is essentially determined by the optical subsystem, then to avoid aliasing the pixel pitch
must satisfy T ≤ 1/2B. For a multi-aperture imaging system consisting of a maximum of ` nonredundant linear
channels we set T = `/2B, undersampling the signal at 1/` the Nyquist rate.
For computer simulation purposes we discretize gj in Equation (4), leading to the following representation in
terms of discrete quantities,
gj = DSj Hj f + ηj ,
(5)
where,
• f is a target representation for the object, stacked as an n2 × 1 vector of high-resolution pixel values,
• Hj is an n2 × n2 matrix blurring operator associated with the optical PSF hj (x) of the jth lenslet,
• Sj is an n2 × n2 matrix representing the translation of the imaging system due to the relative position of
lenslet j with respect to the object and the detector,
• D is an (n2 /`) × n2 decimation matrix√operator√
associated with the detector PSF, transforming the target
from n × n high resolution to the (n/ `) × (n/ `) (low) resolution of the detector,
• gj is the low-resolution discrete image measurements associated with lenslet j stacked as an (n2 /`) × 1
vector and,
• ηj describes a noise process associated with the j th image.
Assuming k ≤ ` lenslets are available for digital super-resolution, the image measurements from Equation (5)
can be arranged together to describe an overall imaging equation of the form,
g = Kf + η,
(6)
where,


S1 H1


..
K = (Ik ⊗ D) 

.
Sk Hk
(7)
is a matrix of size k(n2 /`) × n2 . Equation (6) leads to a least squares inverse problem whose solution is the
desired high resolution reconstruction f .
For simulation purposes, we also adopt a widely-used noise model6 given by:
p
ηj = g̃j ηj1 + σηj2
(8)
where g̃j = DSj Hf . Both ηj1 and ηj2 are standard normally distributed random variables with mean equal to
zero and variance equal to one. p
The parameter σ is the standard deviation of the image-independent noise term.
The normally distributed term g̃j ηj1 in equation (8) approximates the Poisson noise associated with the light
detection process.
2.2. The Inverse Problem
The numerical solution of the inverse problem associated with equation (6) is attracting a good deal of attention
in the image processing community. Most approaches are based on the minimization of a least-squares cost
functional of the form:
2
J(f ) = ||g − Kf ||2 + constraints,
(9)
where constraints are used to enforced desired characteristics in the computed solution, such as smoothness,
sparsity, etc. We note that (9) is a non-trivial computational problem as K is not only ill-conditioned but in
many cases it is also singular when k < ` or when at least two channels contain a redundant amount of subpixel
shift. In practice, we find often that k < ` and the system is under-determined. For values of k reasonably
close to `, acceptable reconstructions can still be obtained. Smaller values of k also reduce the computational
cost of the reconstruction. In addition, full knowledge of matrix K assumes determination of the subpixel shifts
(for Sj ), and the lenslet optical PSFs (for Hj ), which in practice are not always accurately known. These and
other practical issues are dealt within our software component of the PERIODIC imager. Figures 3 and 4 show
some digital super-resolution reconstructions from real data obtained with the PERIODIC prototype shown in
Figure 2. Figure 3(b) shows the digital super-resolution reconstruction of a USAF target using a set of eight
low-resolution sub-images. A typical low-resolution sub-image is shown in Figure 3(a). Figure 4(b) shows marked
improvement of facial features over the low-resolution imagery in Figure 4(a).
(a) Low-resolution subimage with linear interpolation.
(b) Digital super-resolution reconstruction
from a set of eight sub-images.
Figure 3. Digital super-resolution reconstruction of an USAF target.
(a) Low-resolution subimage with linear interpolation.
(b) Digital super-resolution reconstruction
from a set of eight sub-images.
Figure 4. Digital super-resolution reconstruction of facial features.
3. PSF ENGINEERING USING CUBIC PHASE MASKS
In a recent paper, Ashok and Neifeld1 showed that for imagers with pixel-limited resolution an impulse-like
optical PSF maybe suboptimal as a mechanism for encoding object information while tolerating distortions
introduced by the detector array. In such systems, relatively large photodetectors may image two point sources
separated by a subpixel amount onto the same pixel, producing blurring or aliasing in the measured data as
a result of undersampling. Their proposal employs pseudo-random phase masks placed in the aperture stop of
each lenslet in a conventional multi-lens imager to extend the optical PSF over a pixel amount, enabling the
extraction of subpixel position information from the sampled detector outputs.
Here, we explore the use of cubic phase masks in the design of extended optical PSFs for use within a multilens imaging system with pixel-limited resolution. Our approach is motivated by the observation that under
certain conditions carefully designed cubic phase masks can boost the singular values of an imaging operator
above the noise floor. Figure 5 illustrates this assertion where cubic phase masks are shown to improve the
information content when an imager is affected by defocus blur. For the present work we expect that, like
pseudo-random phase masks,1 cubic-type phase masks may increase or precondition the information content
5
10
0
! = 0, " = 0
10
0
! = 20, " = 10
1
! = 60, " = 10
−3
10
! = 0, " = 10
4
3
10
−6
10
r"(!)
5
7
−9
10
−12
3
10
10
9
11
13
15
17
19
−15
10
0
1
2
3
4
5
singular values
6
7
8
9
0
10
5
x 10
(a) Plot of singular values of blurring operator H, showing
how a cubic phase mask (α = 20 or 60) can boost singular
values above a 10− 3 noise floor in the presence of defocus
blur (α = 0, τ = 10).
20
30
40
cubic parameter !
50
60
(b) Sum of singular values of a blurring operator H as
a function of defocus and cubic parameter α (x-axis),
showing that optimal information content can be obtained with a cubic phase mask given a fixed defocus
amount.
Figure 5. Effect of cubic phase masks on the singular value spectrum of blurring matrix operators.
contained within the singular values of the system matrix K in equation (6), leading to increased digital superresolution reconstruction performance.
For the purpose of our simulation study, we consider only 1D objects and image measurements.1 The base
imaging system under consideration is modeled after recent PERIODIC prototypes as having an array of lenslets
of F/# = 3.5 and back focal length of 5 mm. The detector is modeled as having pixels of 7.5 µm in size, with
a full-well capacity of 45,000 electrons and 100% fill factor. The distance from the imager to the plane of best
focus is 1m, giving an airy disk of 4.7 µm or spatial resolution of 2.3 µm. We assume a spectral bandwidth
centered at λ = 550 nm. The undersampling factor for this system is ` = 8.
4. SIMULATION RESULTS
4.1. Rayleigh Resolution Performance
Our simulation experiment is set up as follows. We use equations (1 - 6) to construct low resolution imagery for
both a conventional PERIODIC imager and a multi-lens imager containing identical cubic phase masks in all
lenslets. The base system for both imagers is specified in the previous section. We use both image-dependent
(shot noise) and image-independent (readout noise) noise terms (see equation (8)), with σ = 0.01 max{g˜j }. The
target high-resolution image vector f is of size 560×1. With an undersampling factor of ` = 8, the low-resolution
images gj are of size 70×1. The subpixel shift for each sub-image is chosen from a uniformly random distribution.
We adopt the Rayleigh’s criterion to measure the performance of the conventional PERIODIC imager and
that of the multi-lens imager with a cubic phase mask in the aperture stop. We quantify their trade-off between
resolution and signal-to-noise ratio (SNR) for varying cubic phase mask strength α. To estimate the Rayleigh
resolution we use Ashok and Neifeld’s1 method and image a point-source object to obtain 1 ≤ k ≤ ` lowresolution measurements of the form (4). We then apply the conjugate gradient least squares method (CGLS)
to the aggregate (6), with fixed number of iterations. Note that here we assume full knowledge of the system
optical PSF, which may be obtained in a laboratory setting. The reconstructed point-source is then used to find
the overall PSF of the imager. A diffraction-limited sinc2 PSF is fitted in the least squares sense to the overall
PSF and the Rayleigh spatial resolution (distance from the origin to the first zero-crossing) is computed from it.
The Rayleigh resolution metric is computed and averaged over 30 randomly chosen subpixel shift sets for each
value of k. Figure 6 shows a sample computed overall PSF and its least squares fitted sinc2 PSF.
0.25
computed overall PSF
sinc2 LS fit
0.2
0.15
0.1
0.05
0
−20
−15
−10
−5
0
5
spatial dimension (µm)
10
15
20
Figure 6. Sample computed overall PSF (dashed) and least squares fitted sinc2 PSF. The Rayleigh spatial resolution is
the distance from the origin to the first zero-crossing of the sinc2 PSF.
A large number of cubic phase enhanced multi-lens imagers were simulated and their performance tested
using the Rayleigh spatial resolution criterion. It was quickly evident that cubic phase masks with relative large
steep surfaces (large α value) extended the PSF far too broadly, spreading much of the energy over too many
pixels. In these situations, noise makes it nearly impossible to extract useful subpixel shift information and the
overall performance was above that of a conventional PERIODIC imager. Thus, we see no clear opportunity
for simultaneously improving resolution performance while extending the depth of field. However, resolution
performance was significantly improved with respect to the conventional PERIODIC imager for small α values.
Figure 7 shows the resolution performance of a cubic phase mask enhanced imager, for an optimal value of
α = 1.6, versus that of the conventional PERIODIC imager. This plot shows that using only two frames of a
cubic phase mask enhanced imager can produce imagery of similar quality to that of a conventional PERIODIC
imager of five frames. Moreover, the smallest resolution achievable with the cubic phase mask enhanced imager is
finer, by roughly 0.5µm, than that of the conventional imager. However, at k = 1 frames the conventional system
outperforms the cubic phase mask enhanced imager.∗ Figure 8 shows the extended cubic PSF corresponding to
the resolution performance plot of Figure 7, zoomed over its center region.
4.2. Overall PSF Reconstruction from Low-Resolution PSFs
In this section we study the effectiveness of numerically estimating the overall optical PSF of a multi-lens imager
containing relatively large photodetectors and hence significant undersampling factor. The overall optical PSF
may be accurately estimated in a lab setting, however its reconstruction from low-resolution realizations is of
practical interest. We assume an array of lenslets with identical cubic phase mask on the aperture stop. In this
situation, the overall imaging equation becomes,


S1


g = (Ik ⊗ D)  ...  Hf + η.
(10)
Sk
∗
We suspect this to be a result of using CGLS with a constant, instead of optimal, number of iterations.
9
Conventional imager
Cubic phase enhanced imager
8
spatial resolution (µm)
7
6
5
4
3
2
1
1
2
3
4
5
number of frames (k)
6
7
8
Figure 7. Rayleigh spatial resolution versus number of frames for the conventional PERIODIC imager and multi-lens
imager with a cubic phase mask in the aperture stop.
0.08
Diffraction−limited PSF
Extended Cubic PSF
0.07
0.06
amplitude
0.05
0.04
0.03
0.02
0.01
−18.75
−11.25
−3.75
3.75
spatial dimension (µm)
11.25
18.75
Figure 8. Diffraction-limited versus extended cubic PSF.
Assuming f is a point source then h = Hf is the overall PSF. We thus solve a least squares problem of the form,


J(h) = g − (Ik ⊗ D) 
4
 2
S1

..
h
+ constraints.

.
Sk
(11)
2
We use Chung et al. weighted generalized cross-validation method with Lanczos hybrid regularization (HyBR)
method to optimally minimize J(h). Figure 9 shows a surface plot of the relative error with respect to the true
optical PSF obtained using HyBR, as a function of number of frames k and cubic parameter α. The surface on
the top shows is for multi-lens systems with undersampling factor ` = 11, corresponding to lenses of F/# = 2.5.
The smaller surface below is for multi-lens systems with undersampling factor of ` = 5. As expected, the relative
error decreases with the number of frames. Also notice that a lower undersampling factor leads to increased
accuracy in the determination of the optical PSF. Figure 10 shows a reconstructed PSF for α = 20, and using one
0.9
relative error
0.8
0.7
0.6
60
0.5
40
20
0.4
0
2
4
6
8
10
12
0
cubic parameter (α)
number of frames (k)
Figure 9. Surface plot of relative error in the determination of overall optical PSF from undersampled realizations.
frame (left) and five frames (right). Notice that clearly high resolution features are better reconstructed as the
number of frames k is increased. We note that reconstruction of the cubic PSF is challenging due to its smooth
and rough features, which cannot easily be recovered simultaneously using standard regularization approaches.
0.1
diffraction−limited cubic PSF
reconstructed PSF k=1
0.1
0.09
0.09
0.08
0.08
0.07
0.07
0.06
0.06
0.05
0.05
0.04
0.04
0.03
0.03
0.02
0.02
0.01
0.01
0
−35
−30
−25
−20
−15
−10
−5
spatial dimension (µm)
0
(a) One-frame PSF reconstruction.
5
10
0
−35
diffraction−limited cubic PSF
reconstructed PSF k=5
−30
−25
−20
−15
−10
−5
spatial dimension (µm)
0
5
10
(b) Five-frame PSF reconstruction.
Figure 10. Reconstruction of optical PSF for a cubic phase mask of α = 20.
5. CONCLUSIONS AND FUTURE WORK
We have demonstrated the viability of cubic phase masks for improving the resolution performance of multi-lens
imaging systems. Like pseudo-random phase masks, a weak cubic can produce extended PSFs that facilitate the
extraction of subpixel shift information from the sample detector outputs. Such a cubic phase mask can boost
the singular values of the overall system matrix above the noise floor, improving restorability of the signal. This
process constitutes a form of physical preconditioning that takes place before the signal is discretized by the
detector. While the cubic is not able to improve resolution performance to the extent achieved with pseudorandom phase masks,1 in practice it can additionally help mitigate aberrations and focal length variations
resulting from imperfections in the manufacturing and assembly process.
Other candidate phase masks may be used for improving the resolution performance of multi-lens imaging
systems, beyond that achieved by the cubic and pseudo-random phase masks. The design space is large and
a method for efficiently determining potential candidates is needed. Information theoretic approaches may be
useful in this regard.
6. ACKNOWLEDGMENTS
The research described in this paper was supported in part by grants from the Intelligence Advanced Research
Projects Agency (IARPA) through the Defense Microelectronics Activity (DMEA) under cooperative agreement number H94003-08-2-0802, the Army Research Office (ARO), with award numbers W911NF-05-1-0402 and
DAAD19-00-1-0540, and the Air Force Office of Scientific Research (AFOSR), with award number FA9550-081-0151. Their kind support is sincerely appreciated. The authors also wish to thank our PERIODIC project
collaborators Ryan Barnard, Scott Mathews, Mark Mirotznik, Jim Nagy, and Peter Zhang.
REFERENCES
1. Ashok, A., and Neifeld, M. Pseudorandom phase masks for superresolution imaging from subpixel
shifting. Applied Optics 46, 12 (2007), 2256–2268.
2. Barnard, R., Gray, B., Pauca, V., Torgersen, T., Mirotznik, M., van der Gracht, J., Plemmons, R., Behrmann, G., Mathews, S., and Prasad, S. PERIODIC: State-of-the-art array imaging
technology. In Proc. 2007 ACM Southeast Conference (Winston-Salem, NC, March 2007), ACM, pp. 544–
546.
3. Barnard, R., Pauca, V., Torgersen, T., Plemmons, R., Prasad, S., van der Gracht, J., Nagy,
J., Chung, J., Behrmann, G., Mathews, S., and Mirotznik, M. High-resolution iris image reconstruction from low- resolution imagery. In Proc. SPIE, Advanced Signal Processing Algorithms, Architectures,
and Implementations XVI (San Diego, CA, Aug. 2006), vol. 6313, SPIE, pp. 1–13.
4. Chung, J., Nagy, J., and O’Leary, D. P. A weighted GCV method for Lanczos hybrid regularization.
Elec. Trans. Numer. Anal. 28 (2008), 149–167.
5. DISP. The Duke Imaging and Spectroscopy Program. http://www.disp.duke.edu/projects/imaging,
2006.
6. Jain, A. K. Fundamentals of digital image proccessing. Information and System Sciences. Prentice-Hall
International, 1989.
7. Kitamura, Y., Shogenji, R., Yamada, K., Miyatake, S., Miyamoto, M., Morimoto, T., Masaki,
Y., Kondou, N., Miyazaki, D., Tanida, J., and Ichioka, Y. Reconstruction of a high-resolution image
on a compound-eye image-capturing system. Applied Optics 43, 8 (2004), 1719–1727.
8. Pauca, V., Plemmons, R., Prasad, S., Torgersen, T., and van der Gracht, J. Integrated opticaldigital approaches for enhancing image restoration and focus invariance. In Proc. SPIE, Advanced Signal
Processing Algorithms, Architectures, and Implementations XIII (San Diego, CA, Aug. 2003), vol. 5205,
SPIE, pp. 348–357.
9. PERIODIC. The Practical Enhanced-Resolution Integrated Optical-Digital Imaging Camera (PERIODIC)
Project. http://chroma.cs.wfu.edu, 2008.
10. Prasad, S. Digital superresolution and the generalized sampling theorem. J. Opt. Soc. Am. A 24 (2007),
311–325.
11. Prasad, S., Pauca, V., Plemmons, R., Torgersen, T., and van der Gracht, J. Pupil-phase
optimization for extended-focus aberration-corrected imaging systems. In Proc. SPIE, Advanced Signal
Processing Algorithms, Architectures, and Implementations XIV (Denver, CO, Aug. 2004), vol. 5559, SPIE,
pp. 335–345.