Next-Generation Stanford Shading System
advertisement
Fourier Slice Photography Ren Ng Stanford University Conventional Photograph Light Field Photography Capture the light field inside the camera body Hand-Held Light Field Camera Medium format digital camera Camera in-use 16 megapixel sensor Microlens array Light Field in a Single Exposure Light Field in a Single Exposure Light Field Inside the Camera Body Digital Refocusing Digital Refocusing Questions About Digital Refocusing What is the computational complexity? Are there efficient algorithms? What are the limits on refocusing? How far can we move the focal plane? Overview Fourier Slice Photography Theorem Fourier Refocusing Algorithm Theoretical Limits of Refocusing Previous Work Integral photography Lippmann 1908, Ives 1930 Lots of variants, especially in 3D TV Okoshi 1976, Javidi & Okano 2002 Closest variant is plenoptic camera Adelson & Wang 1992 Fourier analysis of light fields Chai et al. 2000 Refocusing from light fields Isaksen et al. 2000, Stewart et al. 2003 Fourier Slice Photography Theorem In the Fourier domain, a photograph is a 2D slice in the 4D light field. Photographs focused at different depths correspond to 2D slices at different trajectories. Digital Refocusing by Ray-Tracing x u Lens Sensor Digital Refocusing by Ray-Tracing x u Imaginary film Lens Sensor Digital Refocusing by Ray-Tracing x u Imaginary film Lens Sensor Digital Refocusing by Ray-Tracing x u Imaginary film Lens Sensor Digital Refocusing by Ray-Tracing x u Imaginary film Lens Sensor Refocusing as Integral Projection u x x u Imaginary film Lens Sensor Refocusing as Integral Projection u x x u Imaginary film Lens Sensor Refocusing as Integral Projection u x x u Imaginary film Lens Sensor Refocusing as Integral Projection u x x u Imaginary film Lens Sensor Classical Fourier Slice Theorem Integral Projection 1D Fourier Transform 2D Fourier Transform Slicing Classical Fourier Slice Theorem Integral Projection 1D Fourier Transform 2D Fourier Transform Slicing Classical Fourier Slice Theorem Integral Projection 1D Fourier Transform 2D Fourier Transform Slicing Classical Fourier Slice Theorem Integral Projection Spatial Domain Fourier Domain Slicing Classical Fourier Slice Theorem Integral Projection Spatial Domain Fourier Domain Slicing Fourier Slice Photography Theorem Integral Projection Spatial Domain Fourier Domain Slicing Fourier Slice Photography Theorem Integral Projection 4D Fourier Transform Slicing Fourier Slice Photography Theorem Integral Projection 2D Fourier Transform 4D Fourier Transform Slicing Fourier Slice Photography Theorem Integral Projection 2D Fourier Transform 4D Fourier Transform Slicing Fourier Slice Photography Theorem Integral Projection 2D Fourier Transform 4D Fourier Transform Slicing Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing Theorem Limitations Film parallel to lens Everyday camera, not view camera Aperture fully open Closing aperture requires spatial mask Overview Fourier Slice Photography Theorem Fourier Refocusing Algorithm Theoretical Limits of Refocusing Existing Refocusing Algorithms Existing refocusing algorithms are expensive O(N4) All are variants on integral projection Isaksen et al. 2000 Vaish et al. 2004 Levoy et al. 2004 Ng et al. 2005 where light field has N samples in each dimension Refocusing in Spatial Domain Integral Projection 2D Fourier Transform 4D Fourier Transform Slicing Refocusing in Fourier Domain Integral Projection Inverse 2D Fourier Transform 4D Fourier Transform Slicing Refocusing in Fourier Domain Integral Projection Inverse 2D Fourier Transform 4D Fourier Transform Slicing Asymptotic Performance Fourier-domain slicing algorithm Pre-process: O(N4 log N) Refocusing: O(N2 log N) Spatial-domain integration algorithm Refocusing: O(N4) Resampling Filter Choice Triangle filter (quadrilinear) Kaiser-Bessel filter (width 2.5) Gold standard (spatial integration) Overview Fourier Slice Photography Theorem Fourier Refocusing Algorithm Theoretical Limits of Refocusing Problem Statement Assume a light field camera with An f /A lens N x N pixels under each microlens If we compute refocused photographs from these light fields, over what range can we move the focal plane? Analytical assumption Assume band-limited light fields Band-Limited Analysis Band-Limited Analysis Band-width of measured light field Light field shot with camera Band-Limited Analysis Band-Limited Analysis Band-Limited Analysis Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing Results of Band-Limited Analysis Assume a light field camera with An f /A lens N x N pixels under each microlens From its light fields we can Refocus exactly within depth of field of an f /(A•N) lens In our prototype camera Lens is f /4 12 x 12 pixels under each microlens Theoretically refocus within depth of field of an f/48 lens Light Field Photo Gallery Stanford Quad Rodin’s Burghers of Calais Palace of Fine Arts, San Francisco Palace of Fine Arts, San Francisco Waiting to Race Start of the Race Summary of Main Contributions Formal theorem about relationship between light fields and photographs Computational application gives asymptotically fast refocusing algorithm Theoretical application gives analytic solution for limits of refocusing Future Work Apply general signal-processing techniques Cross-fertilization with medical imaging Thanks and Acknowledgments Collaborators on camera tech report Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan Readers and listeners Ravi Ramamoorthi, Kayvon Fatahalian, Brad Osgood, Vaibhav Vaish, Gaurav Garg, Anonymous SIGGRAPH reviewers Brian Curless, Dwight Nishimura, Mike Cammarano, Billy Chen, Jeff Klingner Funding sources NSF, Microsoft Research Fellowship, Stanford Birdseed Grant Questions? “Start of the race”, Stanford University Avery Pool, July 2005
