PROPERTIES AND APPLICATIONS

Outline
 What are light fields
 Acquisition of light fields
 from a 3D scene
 from a real world scene
 Image rendering from light fields
 Changing viewing angle
 Changing the focal plane
 Sampling and reconstruction
 Depth vs spectral support
 Optimal reconstruction
 Analysis of light transport

Outline
 What are light fields
 Acquisition of light fields
 from a 3D scene
 from a real world scene
 Image rendering from light fields
 Changing viewing angle
 Changing the focal plane
 Sampling and reconstruction
 Depth vs spectral support
 Optimal reconstruction
 Analysis of light transport

The Plenoptic Function
Plenus – Complete, full.
Optic - appearance, look.
The set of things one can ever see
𝑷(𝒙, 𝒚, 𝒛, 𝜽, 𝝓, 𝒕, 𝝀)
Light intensity as a function of
◦ Viewpoint – orientation and position
◦ Time
◦ Wavelength
7D function!

The 5D Plenoptic Function
Ignoring wavelength and time
We need a 5D function to describe light rays across occlusions
◦ 2D orientation
◦ 3D position

The Light Field (4D
Assuming no occlusions
◦ Light is constant across rays
◦ Need only 4D to represent the space of Rays
Is this assumption reasonable?
In free space, i.e outside the convex hull of the scene occluders

The Light Field
Parameterizations
◦ Point on a Plane or curved Surface (2D) and Direction on a Hemisphere (2D)
◦ Two Points on a Sphere
◦ Two Points on two different Planes

Two Plane Parameterization
Convenient parameterization for computational photography
Why?
• Similar to camera geometry (i.e. film plane vs lens plane)
• Linear parameterization - easy computations , no trigonometric functions, etc.

2D light field
Used for visualization. Assume the world is flat (2D)

Intuition
The image a pinhole at
(u,v) captures
𝐼(𝑢, 𝑣) = 𝐿(: , : , 𝑢, 𝑣)
𝐿(𝑠, 𝑡, : , : )
All views of a pixel (s,t)
Light Field Rendering , Levoy Hanrahan '96.

Outline
 What are light fields
 Acquisition of light fields
 from a 3D scene
 from a real world scene
 Image rendering from light fields
 Changing viewing angle
 Changing the focal plane
 Sampling and reconstruction
 Depth vs spectral support
 Optimal reconstruction
 Analysis of light transport

Acquisition of Light Fields
Synthetic 3D Scene
◦ Discretize s,t,u,v and capture all rays intersecting the objects using a standard Ray Tracer

Acquisition of Light Fields
Real world scenes
Will be explained in more detail next week…

Outline
 What are light fields
 Acquisition of light fields
 from a 3D scene
 from a real world scene
 Image rendering from light fields
 Changing viewing angle
 Changing the focal plane
 Sampling and reconstruction
 Depth vs spectral support
 Optimal reconstruction
 Analysis of light transport

Changing the View Point
Problem: Computer Graphics
◦ Render a novel view point without expensive ray tracing
Solution:
◦ Sample a Synthetic light field using Ray Tracing
◦ Use the Light Field to generate any point of view, no need to Ray Trace
Light Field Rendering , Levoy Hanrahan '96.

Changing the View Point
Conceptually: Use Ray Trace from all pixels in image plane pinhole
Actually: Use Homographic mapping from XY plane to the VU and TS, and lookup resulting ray radiance.

Light Field Interpolation
𝐼 𝑋, 𝑌 = 𝐿(𝐻
𝑈𝑉 𝑥, 𝑦 , 𝐻
𝑆𝑇
Problem: Finite sampling of the Light Field –
◦ 𝐻
𝑈𝑉 𝑥, 𝑦 , 𝐻
𝑆𝑇 𝑥, 𝑦 may not be sampled 𝑥, 𝑦 )
Solution: Proper interpolation / reconstruction is needed
◦ Nearest neighbor,
◦ Linear,
◦ Custom Filter
Detailed Analysis later on…
NN NN + Linear Linear

Changing the focal plane
Fourier Slice Photography , Ng, 05

In-Camera Light Field Parameterization

The camera operator
Can define a camera as an operator on the Light Field.
◦ The conventional camera operator:
[Stroebel et al. 1986] x y

Reminder - Thin lens formula
1
+
1
D’ D
= const
D
D’
To focus closer - increase the sensor-to-lens distance .

Refocusing - Reparameterization
Type equation here.
𝐿
𝐹
′ 𝑢, 𝑥 = 𝐿
𝐹
(𝑢, 𝑢 + 𝑥 − 𝑢 𝛼
)

Reparametrization - 4D

Refocusing - Reparameterization
Refocus
Change of distance between planes
𝐹
′
= 𝛼𝐹
Reparameterization of the light field
Shearing of the Light field

Refocusing camera operator
Shear and Integrate the original light field
*(cos term from conventional camera model is absorbed into L)

Computation of Refocusing Operator
• Naïve Approach
• For every X,Y go over all U,V and calculate the sum after reparameterization => O(n^4) y
′ ′
′ x
• Can we do better ????

Fourier Slice Theorem
𝐹 ∘ 𝐼 = 𝑆 ∘ 𝐹
• F – Fourier Transform Operator
• I – Integral Projection Operator
• S – Slicing Operator

Fourier Analysis of the Camera Operator
Recall that the Refocusing Camera Operator is:
And from the Last theorem we get The Fourier Slice Photography Theorem
Better Algorithm! 𝑃𝑟𝑒𝑝𝑟𝑜𝑐𝑒𝑠𝑠: 𝑂 𝑛 4 log 𝑛 . 𝐶𝑎𝑙𝑐𝑢𝑙𝑎𝑡𝑒 𝑛𝑒𝑤 𝐷𝑒𝑝𝑡ℎ 𝛼: 𝑂 𝑛 2 + 𝑂(𝑛 2 log 𝑛)

Fourier Slice Photography , Ng, 05

Fourier Slice Photography Thm – More corollaries
Two important results that are worth mentioning:
1. Filtered Light Field Photography Thm
𝑳
𝑭
′ *K=?
2. The light field dimensionality gap

Filtered Light Field Photography Thm
Theorem: Filtered Light Field Photography

The light field dimensionality gap
◦ The light field is 4D
◦ In the frequency domain – The support of all the images with different focus depth is a 3D manifold
This observation was used in order to generate new views of the scene from a focal stack (Levin et al. 2010)

Outline
 What are light fields
 Acquisition of light fields
 from a 3D scene
 from a real world scene
 Image rendering from light fields
 Changing viewing angle
 Changing the focal plane
 Sampling and reconstruction
 Depth vs spectral support
 Optimal reconstruction
 Analysis of light transport

Light Field Sampling
• Light Field Acquisition – Discretization
• Light Field Sampling is Limited
Example – Camera Array: t,s u,v

Sampling in frequency domain
Aliasing in the frequency domain
*
Need to analyze Light Field Spectrum
=

Scene Depth and Light Field
 Light Field Spectrum is related to Scene Depth
 From Lambertian property each point in the scene corresponds to a line in the Light Field
 Line slope is a function of the depth (z) of the point.
Plenoptic Sampling , Chai et al., 00.

Spectral Support of Light Field
Constant Depth
Scene Light Field LF Spectrum
Plenoptic Sampling , Chai et al., 00.

Spectral Support of Light Field
Varying Depth
Scene LF Spectrum
Plenoptic Sampling , Chai et al., 00.

Spectral Support of Light Field
Plenoptic Sampling , Chai et al., 00.

Reconstruction Filters
Optimal Slope for filter:
Plenoptic Sampling , Chai et al., 00.

Limitations
Assumptions
◦ Lambertian surfaces
◦ Free Space – No occlusions

Frequency Analysis of Light Transport
• Informally: Different features of lighting and scene causes different effects in the Frequency
Content
• Blurry Reflections
• Shadow Boundries
High frequency Low frequency
A Frequency analysis of Light Transport , Durand et al. 05.

Frequency Analysis of Light Transport
Look at light transport as a signal processing system.
◦ Light source is the input signal
◦ Interaction are filters / transforms
Source Transport Occlusion Transport Reflection
(BRDF)

Local Light Field
We study the local 4D Light Field around a central Ray during transport
◦ In Spatial Domain
◦ In Frequency Domain
* Local light field offers us the ability to talk about the Spectrum
In a local setting

Local Light Field (2D) Parameterization
The analysis is in flatland, an extension to 4D light field is available x-v parameterization x-Θ parameterization
A Frequency analysis of Light Transport , Durand et al. 05.

Example Scenario
Reflection
A Frequency analysis of Light Transport , Durand et al. 05.

Light Transport – Spatial Domain
Light Propagation  Shear of the local Light Field
◦ No change in slope (v)
◦ Linear change in displacement (X)
+

Light Transport – Frequency Domain
Shear in spatial domain is also a shear in Frequency domain

Occlusion
Spatial domain:
Occlusion  pointwise multiplication in the spatial domain
The incoming light field is multiplied by the binary occlusion function of the occluders.
Frequency domain convolution in the frequency domain:

Occlusion – example

Reflection
We consider planar surfaces * and rotation invariant BRDFs here
What happens when light hits a surface?
1. Multiplication by a cosine term 𝑙 𝑥, 𝜃 = 𝑙 𝑥, 𝜃 cos
+
(𝜃)
2. Mirror Reparameterization around the normal direction 𝑙 𝑥, 𝜃 = 𝑙 𝑥, −𝜃
3. convolution with the BRDF 𝑙 𝑥, 𝜃 = 𝑙 𝑥, 𝜃 ∗ 𝑏𝑟𝑑𝑓(𝜃)
* Similar analysis for curved surfaces is also presented in the paper

Reflection - cosine term
Spatial domain - multiplication:: 𝑙
𝑅′ 𝑥, 𝜃 = 𝑙
𝑅 𝑥, 𝜃 cos
+
(𝜃)
Frequency domain:
𝑙
𝑅′
Ω 𝑥
, Ω 𝜃
Ω 𝑥
, Ω 𝜃
∗ 𝐹(cos
+ 𝜃 )

cosine term example
Incoming
Light field
Light field
After cosine term

Reflection – Mirror reparameterization
Mirror Reparameterization around the normal direction
◦ Using the law of reflection 𝜃 𝑖𝑛
= 𝜃 𝑜𝑢𝑡
◦ 𝑙
𝑅′ 𝑥, 𝜃 = 𝑙
𝑅 𝑥, −𝜃 𝜃 𝑖𝑛 𝜃 𝑜𝑢𝑡
◦
Frequency domain: mirror in the spatial domain => mirror in the frequency domain 𝑙
𝑅′
Ω 𝑥
, Ω 𝜃
= 𝑙
𝑅
Ω 𝑥
, −Ω 𝜃

Reparameterization example
Incoming
Light field
Light field
After reparameterization

𝜋
2
Reflection - BRDF
What is a BRDF?
◦ Bidirectional reflectance distribution function
◦ A function of the incoming ant out going angles 𝜌(𝜃 𝑖𝑛
, 𝜃 𝑜𝑢𝑡
)
◦ Tells us how much “light” comes out at a angle 𝜃 𝑜𝑢𝑡 when illuminating the point from 𝜃 𝑖𝑛
.
◦ Different BRDFs model the reflectance properties of different materials
◦ A lot of BRDFs depend only on the difference between 𝜃 𝑜𝑢𝑡 and the mirror reflection direction: 𝜌 𝜃 𝑖𝑛
, 𝜃 𝑜𝑢𝑡
= 𝑐𝑜𝑛𝑠𝑡 𝜌 𝜃 𝑖𝑛
, 𝜃 𝑜𝑢𝑡
= 𝛿 𝜃 𝑟𝑒𝑓𝑙𝑒𝑐𝑡
− 𝜃 𝑜𝑢𝑡 𝜌 𝜃 𝑖𝑛
, 𝜃 𝑜𝑢𝑡
= 𝑓 𝜃 𝑟𝑒𝑓𝑙𝑒𝑐𝑡
− 𝜃 𝑜𝑢𝑡 𝜋
2 𝜋
2 𝜋
2 𝜋
2 𝜋
2

BRDF Intuition
Assume a Specular BRDF, flat surface and a light source at infinity with angle 𝜃 .
𝜃 𝜃
−𝜃 𝜋
2 𝜋
2
Assume a box BRDF, flat surface and a light source at infinity with angle 𝜃 .
𝜃 * = 𝜃
−𝜃 𝜋
2
−𝑎 𝑎 𝜋
2 x (space) x (space)

Reflection - BRDF
Spatial domain:
The BRDF action on a light field is a convolution with the BRDF function 𝑙
𝑅′
= 𝑙
𝑅
∗ 𝜌(𝜃)
Frequency domain:
Convolution is changed into pointwise multiplication 𝑙
𝑅′
Ω 𝑥
, Ω 𝜃
= 𝑙
𝑅
Ω 𝑥
, Ω 𝜃
𝐹(𝜌(Ω 𝜃
))

BRDF example
Type equation here.
Incoming
Light field
⊗ Light field
After BRDF change
× -