# Slides # Light Fields

PROPERTIES AND APPLICATIONS

Outline  What are light fields  Acquisition of light fields  from a 3D scene  from a real world scene  Image rendering from light fields  Changing viewing angle  Changing the focal plane  Sampling and reconstruction  Depth vs spectral support  Optimal reconstruction  Analysis of light transport

Outline  What are light fields  Acquisition of light fields  from a 3D scene  from a real world scene  Image rendering from light fields  Changing viewing angle  Changing the focal plane  Sampling and reconstruction  Depth vs spectral support  Optimal reconstruction  Analysis of light transport

The Plenoptic Function Plenus – Complete, full. Optic - appearance, look.

The set of things one can ever see 𝑷(𝒙, 𝒚, 𝒛, 𝜽, 𝝓, 𝒕, 𝝀) Light intensity as a function of ◦ Viewpoint – orientation and position ◦ Time ◦ Wavelength 7D function!

The 5D Plenoptic Function Ignoring wavelength and time We need a 5D function to describe light rays across occlusions ◦ 2D orientation ◦ 3D position

The Light Field (4D Assuming no occlusions ◦ Light is constant across rays ◦ Need only 4D to represent the space of Rays Is this assumption reasonable?

In free space, i.e outside the convex hull of the scene occluders

The Light Field Parameterizations ◦ Point on a Plane or curved Surface (2D) and Direction on a Hemisphere (2D) ◦ Two Points on a Sphere ◦ Two Points on two different Planes

Two Plane Parameterization Convenient parameterization for computational photography Why?

• Similar to camera geometry (i.e. film plane vs lens plane) • Linear parameterization - easy computations , no trigonometric functions, etc.

2D light field Used for visualization. Assume the world is flat (2D)

Intuition The image a pinhole at (u,v) captures 𝐼(𝑢, 𝑣) = 𝐿(: , : , 𝑢, 𝑣) 𝐿(𝑠, 𝑡, : , : ) All views of a pixel (s,t) Light Field Rendering , Levoy Hanrahan '96.

Outline  What are light fields  Acquisition of light fields  from a 3D scene  from a real world scene  Image rendering from light fields  Changing viewing angle  Changing the focal plane  Sampling and reconstruction  Depth vs spectral support  Optimal reconstruction  Analysis of light transport

Acquisition of Light Fields Synthetic 3D Scene ◦ Discretize s,t,u,v and capture all rays intersecting the objects using a standard Ray Tracer

Acquisition of Light Fields Real world scenes Will be explained in more detail next week…

Outline  What are light fields  Acquisition of light fields  from a 3D scene  from a real world scene  Image rendering from light fields  Changing viewing angle  Changing the focal plane  Sampling and reconstruction  Depth vs spectral support  Optimal reconstruction  Analysis of light transport

Changing the View Point Problem: Computer Graphics ◦ Render a novel view point without expensive ray tracing Solution: ◦ Sample a Synthetic light field using Ray Tracing ◦ Use the Light Field to generate any point of view, no need to Ray Trace Light Field Rendering , Levoy Hanrahan '96.

Changing the View Point Conceptually: Use Ray Trace from all pixels in image plane pinhole Actually: Use Homographic mapping from XY plane to the VU and TS, and lookup resulting ray radiance.

Light Field Interpolation 𝐼 𝑋, 𝑌 = 𝐿(𝐻 𝑈𝑉 𝑥, 𝑦 , 𝐻 𝑆𝑇 Problem: Finite sampling of the Light Field – ◦ 𝐻 𝑈𝑉 𝑥, 𝑦 , 𝐻 𝑆𝑇 𝑥, 𝑦 may not be sampled 𝑥, 𝑦 ) Solution: Proper interpolation / reconstruction is needed ◦ Nearest neighbor, ◦ Linear, ◦ Custom Filter

Detailed Analysis later on…

NN NN + Linear Linear

Changing the focal plane Fourier Slice Photography , Ng, 05

In-Camera Light Field Parameterization

The camera operator Can define a camera as an operator on the Light Field.

◦ The conventional camera operator: [Stroebel et al. 1986] x y

Reminder - Thin lens formula 1 D’ + 1 D = const

D D’

To focus closer - increase the sensor-to-lens distance .

Refocusing - Reparameterization Type equation here.

𝐿 𝐹 ′ 𝑢, 𝑥 = 𝐿 𝐹 (𝑢, 𝑢 + 𝑥 − 𝑢 𝛼 )

Reparametrization - 4D

Refocusing - Reparameterization Refocus Change of distance between planes 𝐹 ′ = 𝛼𝐹 Reparameterization of the light field Shearing of the Light field

Refocusing camera operator Shear and Integrate the original light field *(cos term from conventional camera model is absorbed into L)

Computation of Refocusing Operator • Naïve Approach • For every X,Y go over all U,V and calculate the sum after reparameterization => O(n^4) y ′ ′ ′ x • Can we do better ????

Fourier Slice Theorem 𝐹 ∘ 𝐼 = 𝑆 ∘ 𝐹 • F – Fourier Transform Operator • I – Integral Projection Operator • S – Slicing Operator

Fourier Analysis of the Camera Operator Recall that the Refocusing Camera Operator is: And from the Last theorem we get The Fourier Slice Photography Theorem

Better Algorithm!

𝑃𝑟𝑒𝑝𝑟𝑜𝑐𝑒𝑠𝑠: 𝑂 𝑛 4 log 𝑛 . 𝐶𝑎𝑙𝑐𝑢𝑙𝑎𝑡𝑒 𝑛𝑒𝑤 𝐷𝑒𝑝𝑡ℎ 𝛼: 𝑂 𝑛 2 + 𝑂(𝑛 2 log 𝑛)

Fourier Slice Photography , Ng, 05

Fourier Slice Photography Thm – More corollaries Two important results that are worth mentioning: 1. Filtered Light Field Photography Thm 𝑳 𝑭 ′

*K=?

2. The light field dimensionality gap

Filtered Light Field Photography Thm Theorem: Filtered Light Field Photography

The light field dimensionality gap ◦ The light field is 4D ◦ In the frequency domain – The support of all the images with different focus depth is a 3D manifold This observation was used in order to generate new views of the scene from a focal stack (Levin et al. 2010)

Outline  What are light fields  Acquisition of light fields  from a 3D scene  from a real world scene  Image rendering from light fields  Changing viewing angle  Changing the focal plane  Sampling and reconstruction  Depth vs spectral support  Optimal reconstruction  Analysis of light transport

Light Field Sampling • Light Field Acquisition – Discretization • Light Field Sampling is Limited Example – Camera Array: t,s u,v

Sampling in frequency domain Aliasing in the frequency domain * Need to analyze Light Field Spectrum =

Scene Depth and Light Field  Light Field Spectrum is related to Scene Depth  From Lambertian property each point in the scene corresponds to a line in the Light Field  Line slope is a function of the depth (z) of the point.

Plenoptic Sampling , Chai et al., 00.

Spectral Support of Light Field Constant Depth Scene Light Field LF Spectrum Plenoptic Sampling , Chai et al., 00.

Spectral Support of Light Field Varying Depth Scene LF Spectrum Plenoptic Sampling , Chai et al., 00.

Spectral Support of Light Field Plenoptic Sampling , Chai et al., 00.

Reconstruction Filters Optimal Slope for filter: Plenoptic Sampling , Chai et al., 00.

Limitations Assumptions ◦ Lambertian surfaces ◦ Free Space – No occlusions

Frequency Analysis of Light Transport • Informally: Different features of lighting and scene causes different effects in the Frequency Content • Blurry Reflections • Shadow Boundries High frequency Low frequency A Frequency analysis of Light Transport , Durand et al. 05.

# Not Wave Optics!!!

Frequency Analysis of Light Transport Look at light transport as a signal processing system.

◦ Light source is the input signal ◦ Interaction are filters / transforms Source Transport Occlusion Transport Reflection (BRDF)

Local Light Field We study the local 4D Light Field around a central Ray during transport ◦ In Spatial Domain ◦ In Frequency Domain * Local light field offers us the ability to talk about the Spectrum In a local setting

Local Light Field (2D) Parameterization The analysis is in flatland, an extension to 4D light field is available x-v parameterization x-Θ parameterization A Frequency analysis of Light Transport , Durand et al. 05.

Example Scenario Reflection A Frequency analysis of Light Transport , Durand et al. 05.

Light Transport – Spatial Domain Light Propagation  Shear of the local Light Field ◦ No change in slope (v) ◦ Linear change in displacement (X) +

Light Transport – Frequency Domain Shear in spatial domain is also a shear in Frequency domain

Occlusion Spatial domain: Occlusion  pointwise multiplication in the spatial domain The incoming light field is multiplied by the binary occlusion function of the occluders. Frequency domain convolution in the frequency domain:

Occlusion – example

Reflection We consider planar surfaces * and rotation invariant BRDFs here What happens when light hits a surface?

1. Multiplication by a cosine term 𝑙 𝑥, 𝜃 = 𝑙 𝑥, 𝜃 cos + (𝜃) 2. Mirror Reparameterization around the normal direction 𝑙 𝑥, 𝜃 = 𝑙 𝑥, −𝜃 3. convolution with the BRDF 𝑙 𝑥, 𝜃 = 𝑙 𝑥, 𝜃 ∗ 𝑏𝑟𝑑𝑓(𝜃) * Similar analysis for curved surfaces is also presented in the paper

Reflection - cosine term Spatial domain - multiplication:: 𝑙 𝑅′ 𝑥, 𝜃 = 𝑙 𝑅 𝑥, 𝜃 cos + (𝜃) Frequency domain: 𝑙 𝑅′ Ω 𝑥 , Ω 𝜃 Ω 𝑥 , Ω 𝜃 ∗ 𝐹(cos + 𝜃 )

cosine term example Incoming Light field Light field After cosine term

Reflection – Mirror reparameterization Mirror Reparameterization around the normal direction ◦ Using the law of reflection 𝜃 𝑖𝑛 = 𝜃 𝑜𝑢𝑡 ◦ 𝑙 𝑅′ 𝑥, 𝜃 = 𝑙 𝑅 𝑥, −𝜃 𝜃 𝑖𝑛 𝜃 𝑜𝑢𝑡 ◦ Frequency domain: mirror in the spatial domain => mirror in the frequency domain 𝑙 𝑅′ Ω 𝑥 , Ω 𝜃 = 𝑙 𝑅 Ω 𝑥 , −Ω 𝜃

Reparameterization example Incoming Light field Light field After reparameterization

𝜋 2 Reflection - BRDF What is a BRDF?

◦ Bidirectional reflectance distribution function ◦ A function of the incoming ant out going angles 𝜌(𝜃 𝑖𝑛 , 𝜃 𝑜𝑢𝑡 ) ◦ Tells us how much “light” comes out at a angle 𝜃 𝑜𝑢𝑡 when illuminating the point from 𝜃 𝑖𝑛 .

◦ Different BRDFs model the reflectance properties of different materials ◦ A lot of BRDFs depend only on the difference between 𝜃 𝑜𝑢𝑡 and the mirror reflection direction: 𝜌 𝜃 𝑖𝑛 , 𝜃 𝑜𝑢𝑡 = 𝑐𝑜𝑛𝑠𝑡 𝜌 𝜃 𝑖𝑛 , 𝜃 𝑜𝑢𝑡 = 𝛿 𝜃 𝑟𝑒𝑓𝑙𝑒𝑐𝑡 − 𝜃 𝑜𝑢𝑡 𝜌 𝜃 𝑖𝑛 , 𝜃 𝑜𝑢𝑡 = 𝑓 𝜃 𝑟𝑒𝑓𝑙𝑒𝑐𝑡 − 𝜃 𝑜𝑢𝑡 𝜋 2 𝜋 2 𝜋 2 𝜋 2 𝜋 2

BRDF Intuition Assume a Specular BRDF, flat surface and a light source at infinity with angle 𝜃 .

𝜃 𝜃 −𝜃 𝜋 2 𝜋 2 Assume a box BRDF, flat surface and a light source at infinity with angle 𝜃 .

𝜃 * 𝜋 2 −𝑎 𝑎 𝜋 2 = 𝜃 −𝜃 x (space) x (space)

Reflection - BRDF Spatial domain: The BRDF action on a light field is a convolution with the BRDF function 𝑙 𝑅′ = 𝑙 𝑅 ∗ 𝜌(𝜃) Frequency domain: Convolution is changed into pointwise multiplication 𝑙 𝑅′ Ω 𝑥 , Ω 𝜃 = 𝑙 𝑅 Ω 𝑥 , Ω 𝜃 𝐹(𝜌(Ω 𝜃 ))

BRDF example Type equation here.

Incoming Light field ⊗ Light field After BRDF change ×

-