Plenoptic Sampling
Jian Huang, CS 594, Spring 2002
Light Field Methods
• For image-based rendering algorithms based on
plenoptic function, some way of discrete sampling
is always used
• What is the optimal sampling? Can Nyquist
sampling theory be applied?
• Plenoptic Sampling, Proc. SIGGRAPH’2000, JinXiang Chai, Xin Tong, Shing-Chow Chan, HeungYeung Shum, Microsoft Research, China
– Minimal sampling rate for light fields
Background
• Texture mapping, very accurate geometry and only
a few images
• Image-based methods with depths:
– 3D warping, LDI (layered depth image), view morph,
interpolation
– A few images with depth information
• Light field methods
– Do not assume any depth information: light fields,
lumigraph, concentric mosaics
– Rely on over-sampling (large amount images)
– Intensive data acquisition, storage and processing
Two Plane Parametrization
Focal plane (st)
Camera plane (uv)
Object
Reconstruction
•
Plenoptic Sampling
• How many samples of the plenoptic function (e.g.,
from a 4D light field) and how much geometrical
and textural information are needed to generate a
continuous representation of the plenoptic
function?
• Two specific goals:
– Minimum sampling rate for light field rendering
– Minimum sampling curve in joint image and geometry
space
Formulation
• A high-dimension signal processing
problem
• Assumption
– Lambertian surface
– Uniform sampling geometry or lattice
• Study:
– Spectral support of light field signals
– Not closed-form spectral representation
A Key Concept
• The spectral support of a light field signal is bounded by
only the minimum and maximum depths, irrespective of
how complicated the spectral support might be because of
depth variations in the scene.
• Given the minimum and maximum depths, a
reconstruction filter with an optimal and constant depth
can be designed to achieve anti-aliased light field
rendering.
Contribution
• The minimum sampling rate of light field rendering is
obtained by compacting the replicas of the spectral support of
the sampled light field within the smallest interval without
any overlap.
• Using more depth information, plenoptic sampling in the joint
image and geometry space allows us to greatly reduce the
number of images needed.
• The relationship between the number of images and the
geometrical information under a given rendering resolution
can be described by a minimum sampling curve.
• This minimal sampling curve serves as the design principles
for IBR systems, bridging the gap between image-based
rendering and traditional geometry-based rendering.
Convolution
• Sampling a continuous light field function, l, with
sampling pattern, p, reconstruction filter (kernel),
r, reproducing images i.
Spectral Support
• Let z(u,v,s,t) to be the depth function, i.e. geometry
• The same point is views in camera 0 and t as point v and v’.
• Assuming Lambertian surface, each epi-polar line is of
uniform color
Radiance
• The radiance received at camera location (s,t) is:
• With the Fourier transform being: (complicated to
compute)
Spectral Support
• Using a rectangular sampling lattice, the sampled function:
• L has to be bandlimited.
• Need to use a sampling rate higher than Nyquist for alias-free
sampling.
Research Question
• Is there an optimal reconstruction function (like
sinc in conventional signal processing)?
• The design of such a function is related to the
depth function. How exact should depth be
captured and recovered?
Scene with constant depth
• Choosing the first frame as reference, l(u,v,0,0).
Spectral Support
• Focusing on sampling in (v,t) sub-space
Spatially varying depth model
• Straightforward to observe that the spectral
support of a scene of depth range [zmin, zmax] is
bounded by two lines:
Scene Images
Reconstruction at constant depth
• Assuming a constant depth in reconstruction
• Optimal depth:
Sample Renderings
• At min depth, optimal depth, average depth
and max depth:
Minimum Sampling Rate
• How tight can you pack:
Joint Image and Geometry Space
• With a precise geometry, can decompose a scene
into multiple layers of depths
Joint Image and Geometry Space
• With depth uncertainty, for instance, a noisy depth image
Obviously, the curve will be scene dependent!
Results (more in the paper)