General Linear Cameras: Theory and Applications

General Linear Cameras: Theory and Applications by Jingyi Yu Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the MASSACHUSETTS INSTITUTE 0 F TECHNOLOGY August 2005 ) Massachusetts Institute of Technology 2005. All rights reserved. Author............. ............. ........................... Department of Electrical Engineering and Computer Science August 9, 2005 Certified by ........ Certified b y. ~. ... , //~~... ... '' Fredo Durand Assistant Professor Thesis Supervisor . Leonard McMillan Associate Professor · .~- .'(.,y~.....Thesis, Supervisor Acceptedby....... Arthur C. Smith Chairman, Department Committee on Graduate Students MASSACHUSETTS INSITI"E' OF TECHNOLOGY MAR 2 8 2006 I IRPAPIPQ ARCHVES 2 General Linear Cameras: Theory and Applications by Jingyi Yu Submitted to the Department of Electrical Engineering and Computer Science on August 9, 2005, in partial fulfillment of the requirements for the degree of Doctor of Philosophy Abstract I present a General Linear Camera (GLC) model that unifies many previous camera models into a single representation. The GLC model describes all perspective (pinhole), orthographic, and many multiperspective (including pushbroom and two-slit) cameras, as well as epipolar plane images. It also includes three new and previously unexplored multiperspective linear cameras. The GLC model is general and linear in the sense that, given any vector space where rays are represented as points, it describes all 2D affine subspaces (planes) formed by the affine combination of 3 rays. I also present theories of projection and collineation for GLCs and use these theories to explain various multiperspective distortions. Given an arbitrary multiperspective imaging system that captures smoothly varying set of rays, I show how to map the rays onto a 2D ray manifold embedded into a 4D linear vector space. The GLC model can then be use to analyze the tangent planes on this manifold. Geometric structures associated with the local GLC model of each tangent plane provide an intuitive physical interpretation of the imaging system, and they are closely related to the caustics of reflected rays. These geometric structures are characteristic of only 4 of the 8 GLC types. I also prove that the local GLC type at each tangent plane is invariant to the choice of parametrization, and, thus, an intrinsic property of the reflecting surface. Using GLCs to analyze the caustics of reflection extends the previous Jacobian-based approaches, which consider only a pinhole model at each infinitesimal region about each surface point. Finally, I demonstrate how to use the GLC model in computer vision, computer graphics, and optical design applications. In particular, I show how to use GLCs for modelling and rendering multiperspective images and characterizing real multiperspective imaging systems such as catadioptric mirrors. Thesis Supervisor: Fredo Durand Title: Assistant Professor Thesis Supervisor: Leonard McMillan Title: Associate Professor 3 4 Acknowledgments I would like to thank my advisor, Professor Leonard McMillan for his support and advice. It has been a great pleasure working with Leonard on such an interesting and challenging project for the last five years. Thanks for keeping me focused on the big picture and pushing me to do my best. I have learned much more than research from Leonard. Thanks for tolerating my stubbornness from time to time. I would also like to thank my thesis co-advisor, Professor Fr6do Durand, who has made tremendous efforts for helping me to finish my dissertation in time. And also thanks to Fredo for providing the guidelines to the thesis and for assisting me on my job talk. I would like to thank my thesis committee members Professor Steven Gortler and Professor Tomas Lozano-Perez for their time and helpful feedback. To Tomas for his critical efforts working with Fredo and Leonard to make my Ph.D. defense possible. To Shlomo for the useful phone discussions and his recommendation letter to my faculty application. Many thanks to my friends Jason Yang, Jingdan Zhang, Eric Bennett, and Xianfeng Gu. Thanks to Jason for his codes on GPU-based real-time light field renderer. Thanks to Jingdan for his useful inputs on many of my published papers. Thanks to Eric for his outstanding proofreading ability. Thanks to my buddy Xianfeng for his critical help on the "Non-existence of Cross-slit Mirror" proof. Special thanks to Professor Eric Grimson and Professor Srini Devedas for their support during my last semester at MIT. Thanks to Marilyn Pierce, Bryt Bradley, and Missy Wood for handling numerous administrative issues. Thanks to everyone in the MIT Graphics Group for making the lab a fun and productive place to work. Thanks to Professor Marc Pollefeys and Professor Stephen Pizer at UNC Chapel Hill for the useful discussions and their recommendation letters. Thanks to Dr. Harry Shum for his continuous support as a mentor and a friend. I thank my parents for supporting my decision to study in the US. They have been the greatest parents and I love them very much. Without their support, I wouldn't have fulfilled my dream today. Thanks to my buddies, Zhongji Liao and Jingshen 5 Jiang, for their ability and willingness to help during any crisis, large or small, real or imagined. 6 Contents 1 Introduction 1.1 19 Motivation ... 1.2 Thesis Statement 1.3 Contributions . . 1.4 Thesis Overview. .......................... .......................... .......................... .......................... 2 Background and Previous Work . . 2.2 Multiperspective Images ......... . . . . . 2.3 Multiperspective Cameras ....... . . .. 2.3.1 Commonly Used Multiperspective Cameras 2.3.2 Catadioptric Mirrors ...... . . . . . . 2.4 Ray Representation ........... 2.4.1 Light Fields Parametrization . . . . . 2.4.2 Ray Space Analysis ........ . . . .. . ........... ........... ........... ........... ........... ........... ........... ........... 3 General Linear Cameras General Linear Camera Model ...... 3.2 Equivalence of Classic Camera Models . 3.3 Characteristic Equation of GLC ..... Characterizing Classic Camera Models A = 0 case ............. 3.4.2 A Ocase. 22 23 25 27 29 30 32 32 33 34 37 3.1 3.4.1 21 25 2.1 Cameras .................. 3.4 20 ............ 7 . ................ ................ ................ ................ ................ ................ 37 40 42 45 45 46 3.5 3.6 New Multiperspective Camera Models . . . . . . . . . . . . . . .47 3.5.1 New Camera Models ..... . . . . . . . . . . . . . . 49 3.5.2 All General Linear Cameras . . . . . . . . . . . . . . 50 . . . . . . . . . . . . . . 51 Example GLC Images . . ......... 4 GLC Characteristics 55 4.1 Relative Two Plane Parametrization . . . . . . . . . . . . . . 55 4.2 Sweeping Area ............. . . . . . . . . . . . . . . 57 4.3 GLC Slits . . . . . . . . . . . . . . 58 4.4 Linear Ray Manifolds .......... . . . . . . . . . . . . . . 60 4.4.1 Canonical GLC Representation . . . . . . . . . . . . . . 60 4.4.2 GLC to Linear Constraints . . . . . . . . . . . . . . . . . .60 4.4.3 GLC Degrees of Freedom . . . . . . . . . . . . . . 63 4.5 Plfcker Coordinates .......... . . . . . . . . . . . . . . 64 4.6 . . . . . . . . . . . . . . 68 ................ .... Discussion ................ 5 GLC Projection and Collineation 5.1 GLC Projection ................ 5.2 69 69 5.1.1 Plane Sweeping ........... 71 5.1.2 Singularities .............. 72 5.1.3 Projections of Lines .......... 74 5.1.4 Projections of Points at Infinity . . . 75 GLC Collineation ............... 77 5.2.1 General Collineation between GLCs . 80 5.2.2 Collineation Degeneracies ..... 81 6 GLC Differential Invariance Theorem 85 6.1 Limitations of 2PP ............................ 85 6.2 87 Local Tangent GLC ............................ 6.3 Differential Invariant Theorem ...................... 8 88 7 Multiperspective Rendering 93 7.1 Background and Previous Approaches. . . . . . . . . . . . . . . . . . 7.1.1 7.2 7.3 GLC as Multiperspective Images Multiperspective Ray Manifold .... ......... 7.2.1 Piecewise Multiperspective Image Plane 7.2.2 Adjacency Table ............ 93 . . . . 94 . . . . 95 .. . . 97 . .. . 99 . . . . . . . . . . . . . . . .... .... 100 100 . . . . . . . . . . . . . . . .... 102 GLC Multiperspective Rendering 7.3.1 Image Layout 7.3.2 Ray Blending ........ 7.4 Multiperspective Sampling ........... .. .. 103 .. 106 7.4.1 Perspective and Projective Distortions .. 7.4.2 Resampling . .............. . ... 7.5 Multiperspective Rendering Results ....... 7.5.1 Multiperspective Panoramas ...... 7.5.2 Artistic Multiperspective ........ 7.5.3 Multiperspective Faux-Animation . .. 106 .. .... .. 107 107 .. 108 109 .. 8 Reflection Analysis 113 8.1 Background and Previous Work ......... . . . . . . . . 113 ... 8.2 GLC Reflection Analysis ............. . . . . . . . . 115 ... 8.2.1 Multiperspective Reflection Model .... . . . . . . . . 115 ... 8.2.2 Orthographic Viewing Conditions .... . . . . . . . . 118 ... 8.2.3 Perspective Viewing Conditions ..... . . . . . . . . 120 ... Caustic Surfaces ................. . . . . . . . . 125 ... 8.3.1 . . . . . . . . 126 ... . . . . . . . . 129 ... 8.3 Cross-slit Rulings ............ 8.4 Characteristic GLC Distortions ......... 8.4.1 Cross-slit Distortion .......... . . . . . . . . 130 ... 8.4.2 Predicting Distortions on Mesh Surfaces . . . . . . . . 131 ... 9 Conclusions and Future Work 133 9.1 The Theory of GLC ........................... 9 133 9.2 GLC Applications 9.3 Future Work ........................ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 136 A Non-existence of Cross-slit Mirrors 10 139 List of Figures 2-1 Multiperspective images in art. (a) Hall City by M.C. Escher. Portrait of a Young Girl by Pablo Picasso. (b) (c) Red Fish by Henri Matisse ................................... 27 2-2 Pushbroom and Cross-slit Camera. (a) Acquisition geometry of a pushbroom camera. (b) The stationary column sampling routine for synthesizing a pushbroom camera. (c) Acquisition geometry of a cross-slit camera. (d) The linearly varying column sampling routine for synthesizing a cross-slit camera ................................. 2-3 Multiperspective 30 images in graphics and vision. (a) and (b) Images captured from catadioptric mirrors (Courtesy of Tomas Pajdla). (c) A reflection image observed on the car window. (d)A pushbroom panorama by University of Washington (Courtesy of Steve Seitz). 2-4 . An camera array samples a light field with an array of images ..... 2-5 Two plane parametrization. 31 33 A new pinhole image can be synthesized by querying and blending rays from the light field. Each ray is parameterized by intersecting the two parallel planes as [s, t] and [u, v]. 3-1 General Linear Camera Model. a)A GLC is characterized 34 by three rays originated from the image plane. b)It collects all possible affine combination of three rays ......................... 3-2 38 Classic camera models represented as GLC. (a)Two similar triangles on two planes define a pinhole camera; (b)Two parallel congruent triangles define an orthographic camera; (c)Three rays from an cross-slit camera. 11 40 3-3 Bilinear Surfaces. (a) r 3 is parallel to S; (b) r3 is parallel to S, but still intersects S; (c) r 3 is not parallel to S, and does not intersect S either .................................... 48 3-4 Pushbroom, Twisted Orthographic, and Pencil Cameras. (a) A pushbroom camera collects rays on a set of parallel planes passing through a line; (b) A twisted orthographic camera collects rays with uniform directions on a set of parallel planes; (c) A pencil camera collects rays on a set of non-parallel planes that share a line................. 3-5 . 49 Comparison between synthetic GLC images. From left to right, top row: a pinhole, an orthographic and an EPI; middle row: a pushbroom, a pencil and an twisted orthographic; bottom row: a bilinear and an cross-slit ............................................. 3-6 52 GLC images created from a light field. Top row: a pencil, bilinear, and pushbroom image. Bottom row: an cross-slit, twisted orthographic, and orthographic image................................... 3-7 . 52 A multiperspective bilinear GLC image synthesized from three pinhole cameras shown on the right. The generator rays are highlighted in red. 53 3-8 Synthesizing GLCs by cutting through a 3D horizontal light fields. Left: we cut through the light field vertical diagonally to generate a cross-slit GLC. Right: we cut through the light field vertical diagonally to generate a pencil GLC......................... 4-1 53 From left to right: the area formed by the three generators at each sweeping plane. Green corresponds to positive areas and red corresponds to negative. When the area is zero, all rays will pass through a slit on the corresponding sweeping plane................................ 57 12 4-2 Area-Depth Curve (ADC) for GLCs. From left to right: ADC of a pinhole/pencil camera is quadratic and intersects area = 0 axis once; ADC of orthographic/twisted orthographic camera is a constant line parallel to area = 0 axis; ADC of a pushbroom camera is a line that intersects area = 0 axis once; ADC of a cross-slit camera is quadratic and intersects area = 0 axis twice; ADC of a bilinear camera is quadratic and never intersects area = 0 axis; ADC of an EPI camera is the area = 0 axis ................................... 5-1 58 (a) Projecting a point P to a ray in the GLC. (b) The projection of P can be computed using the same affine coordinate on the sweeping plane IIz. . 5-2 71 (a) and (c) are cross-slits images synthesized from light fields. (b) and (d) are pushbroom images synthesized from light fields. The distortions of the curved isolines on the objects illustrate various multi-perspective effects of GLC cameras. The pushbroom camera also exhibits apparent stretching/duplications of objects faraway from the image plane, e.g., the lamp on the background ............................... 5-3 73 (a) Imaging the GLC on a plane parallel to the uv plane. (b) Imaging the GLC on a different plane. (c) A GLC image of a toy dog synthesized from a video using image plane (a). (d) A different image of the same GLC camera using image plane (b) ...................................... 7-1 . 78 Image fusion by composing a GLC image: a new pose of the cow (right) is synthesized by specifying the three generator rays (red) of a GLC from left three poses ....................................... 95 7-2 A neocubism styled multiperspective image synthesized using our framework. It combines pinhole images from different points of views with various multiperspective (pushbroom and cross-slit) images..... 13 . 96 7-3 General Linear Camera Models. There are precisely eight GLCs, shown in (a) to (h). (a) In a pinhole camera, all rays pass through a single point. (b) In an orthographic camera, all rays are parallel. (c) In a pushbroom, all rays lie on a set of parallel planes and pass through a line. (d) In a cross slit camera, all rays pass through two non-coplanar lines. (e) In a pencil camera, all coplanar rays originate from a point on a line and lie on a specific plane through the line. (f) In a twisted orthographic camera, all rays lie on parallel twisted planes and no rays intersect. (g) In an bilinear camera, no two rays are coplanar and no two rays intersect. (h) In an EPI camera, all rays lie on a 2D plane . ................... ......... . 97 7-4 A tessellation of the image plane with triangles that correspond to different types of general linear cameras. This model is equivalent to a 2D piecewise planar manifold embedded in 4D ........... 7-5 .... . Desirable image fragments from different GLCs are overlayed on the im age plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-6 101 Ray coordinate of each vertex in the overlapped region is computed using Algorithm 1............................. 7-7 98 (a) Sampling of a pinhole camera (homography). 103 (b) Sampling of a cross-slit camera. (c) Sampling of an arbitrary multiperspective image is achieved by assigning a sampling plane to each GLC ......... 7-8 104 (a) A multiperspective image before sampling correction; (b) A multiperspective image after sampling correction using sampling planes is applied. In the original stuffed toy, the head is oriented such that it tive im age . looks at the tail. This multiperspective image appears like a plausible perspective image with a different pose of the toy ............ 7-9 (a) Nusch Eluard by Pablo Picasso; (b) Our synthesized multiperspec. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 10 107 7-10 Multiperspective panoramas: (a) Our synthesized multiperspective panorama from a virtual flight path. (b) Blended camera types of the multiperspective image. Different GLCs are marked with different colors, red for cross-slit, blue for pinhole, and green for pushbroom ......... . 110 7-11 (a) A perspective view. (b) A synthesized multiperspective image that fuses different perspective views of the handle and beak. (c) A omniperspective image that shows a 360 degree view of the teapot ..... 7-12 From left to right: extracted images from a faux-animation. 111 The source images were acquired by rotating a ceramic figure on a turn table. Mul- tiperspective renderings were used to turn the head and hind quarters of the figure in a fake image-based animation. sampling. 8-1 Bottom right: with resampling . Top row: without re- . . . . . . . . . . . . . . 111 A reflected image seen on a car window. Subregions illustrate multiperspective distortions such as shearing (yellow), uneven aspect ratio (red), and duplications (green).......................................... 8-2 At each point on the surface, the reflected ray is mapped into the ray space by intersecting with the two parametrization planes ............. 8-3 114 115 (a) If viewed from an orthographic camera, the parametrization plane is perpendicular to the incident direction. (b) If viewed from a pinhole camera, the parametrization plane at each point on the surface is parallel to its tangent plane . 8-4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 The caustic surface and its ruling of typical mirrors. The viewpoint is shown as a blue sphere. (a) A hyperbolic mirror. (c) A parabolic mirror. (e) A cylindrical mirror. (b), (d), and (f) show the ruling of the caustics of (a), (c), (e), respectively. 8-5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Aspect ratio distortion. Top row: reflected images of a hyperbolic, spherical, and cylinder mirror. Bottom row: estimated ratio distortion by the slits distance of the respective mirrors ...................... 15 129 8-6 Estimating reflection distortions of a general surface. (a) A pear-shaped mesh. (b) Estimated distortions. Bright colors correspond to large distortions (closer to pushbroom) and dark colors correspond to small distortions (closer to pinhole). (c) A ray-traced image of the parametric surface. No- tice reflections at regions around the red dashed line correspond to brightest regions in (b) ............................................ . 132 A-1 Reflection between pinhole cameras. Left: an ellipsoid mirror reflects between two pinhole cameras on the same side. Right: a hyperbolic mirror reflects between two pinhole cameras on different sides .... 140 A-2 Geometric constraints on mapping between pinhole and cross-slit camera. 141 16 List of Tables 3.1 Characterize General Linear Cameras by Characteristic Equation 4.1 Characterize GLC with Lines 4.2 Degree of Freedom of GLCs 7.1 Adjacency Tables of GLCs ........................ ...................... ....................... 17 . . 50 59 64 100 18 Chapter 1 Introduction Camera models are fundamental to the fields of computer vision, computer graphics, and photogrammetry. The classic pinhole and orthographic camera models have long served as the workhorse of 3D imaging applications. However, recent develop- ments have suggested alternative multiperspective camera models can provide different and potentially advantageous imaging systems for understanding the structure of observed scenes. Researchers have also shown that these multiperspective are amenable to stereo analysis and interpretation cameras [30, 27, 47]. In contrast to pinhole and orthographic cameras, which can be completely characterized using a simple linear model (the classic 3 by 4 matrix [15]), multiperspective cameras models are defined less precisely. In practice, multiperspective cameras models are described by constructions. By this I mean that a system or process is described for generating each specific class. While such physical models are useful for both acquisition and imparting intuition, they are not particularly amenable to analysis. In this thesis, I present a new theory called the General Linear Camera (GLC) model that unifies many previous multiperspective camera models as well as the pinhole camera into a single representation. The GLC model describes all perspective (pinhole), orthographic, and many multiperspective (including pushbroom and crossslit) cameras, as well as epipolar plane images (EPI). The GLC model is both general and linear in the sense that, given any vector space where rays are represented as 19 points, it describes all 2D affine subspaces (planes) formed by the affine combination of 3 rays. The incident radiance seen along the rays of these 2D affine subspaces are a precise definition of a projected image of a 3D scene. The GLC model and the GLC images help to explain the cause of various multiperspective distortions. show that the GLC model provides an intuitive physical interpretation I further and is closely related to the caustics of the rays. Applications of the GLC model include modelling and rendering multiperspective images, creating cel-animation, image-based faux animation, and characterizing real image systems such as catadioptric mirrors. 1.1 Motivation A perspective image represents the spatial relationships of objects in a scene as they would appear from a single viewpoint. In contrast, a multiperspective image combines what is seen from several viewpoints into a single image. Despite their incongruity of view, effective multiperspective images are still able to preserve spatial coherence. More importantly multiperspective images can depict, within a single context, details of a scene that are simultaneously inaccessible from a single view, yet easily interpretable by a viewer. Multiperspective rendering techniques are frequently employed by artists to depict more than can be seen from any specific point. Classic examples include the visual paradoxes of Escher, and the Cubism of Picasso and Braque. Multiperspective images have also received attention from the computer vision community for analyzing structure revealedvia motion [30,27] and generating panoramic images with a wide field-of-view using mirrors [22]. Several researchers have proposed alternative multiperspective camera models which capture rays from different points in space. These multiperspective cameras include pushbroom cameras [13], which collect rays along parallel planes from points swept along a linear trajectory, and two-slit cameras [26], which collect all rays passing through two lines. Finally in our everyday lives, we experience multiperspective images when observing reflections off of curved surfaces. Despite all these efforts in modelling and rendering multiperspective images, there 20 remain several fundamental problems unresolved. First, most of the previous multiperspective models are constructed by enforcing specific geometric constraints. This makes it difficult to characterize and study the general properties and variations of multiperspective cameras. A unified model is desirable to model, render, and characterize arbitrary multiperspective images. Second, very little work has been to qualitatively and quantitatively analyze multiperspective distortions, such as curving of the lines, shearing of the angles, stretching of scene geometry, and the duplication of the scene points. Most previous studies have been restricted to specific mirrors and camera types. Furthermore, there has been little work on that relates image distortions to the geometry of the imaging system. Finally, only perspective rendering is widely supported in three-dimensional computer graphics. There are no simple and efficient methods for supporting multiperspective rendering. This is partly due to the inherent difficulty of specifying the projection process necessary for mapping 3 dimensions down to 2 for multiperspective renderings. This thesis addresses these problems by presenting a unified framework of the General Linear Camera model. It provides a complete set of analytical and modelling tools for interactively designing, analyzing, and rendering multiperspective images. 1.2 Thesis Statement The General Linear Camera (GLC) model unifies many previous perspective and multiperspective camera models and provides a complete description of all 2D aine subspaces of rays. The GLC model also provides a first-order differential tool for modelling, rendering, and analyzing arbitrary (higher-order) multiperspective imaging systems. The GLC model unifies perspective (pinhole), orthographic, and many multiperspective (including pushbroom and two-slit) cameras, as well as epipolar plane images into a single analytical representation. GLC projections and collineations can be used to model any coherent multiperspective imaging systems. Geometrically, the GLC model characterizes all possible all 2D affine subspaces of rays. These set of 21 hyperplanes provide a first-order approximation tool to analyze arbitrary multiperspective imaging systems. I show the GLC model has many useful applications in computer graphics and computer vision, including creating multiperspective panoramas, neocubist-style renderings, faux animations from still-life scenes, characterizing multiperspective image distortions, and guiding mirror designs to improve catadioptric imaging systems. 1.3 Contributions This thesis makes the followingcontributions * It introduces a General Linear Camera (GLC) model that unifies many previous perspective and multiperspective cameras within a single framework and discovers three new, previously unexplored multiperspective camera types. * It provides an analytical representation of the GLC model as 2D affine subspaces of rays parameterized under two-plane-parametrization. * It introduces a set of characteristic equations for identifying each of the 8 atomic GLC types, and proves that these types are completely characterize all possible linear multiperspective camera models. * It provides a projection and collineation theory to model the GLC image distortions. * It describes an analytical framework for mapping arbitrary imaging systems into 2D ray manifolds and uses GLCs as the first-order differential model of local ray behavior. * It proves a Differential Invariant Theorem that shows the local GLC model on the ray manifold is invariant to choice of the parametrization planes. * It provides a complete set of adjacency rules for constructing piecewise linear ray manifolds by tiling GLCs onto an image plane. 22 * It provides an analysis framework for studying reflection ray manifolds via local GLC models. It proves that local reflections observed on arbitrary mirrors can originate from only four of the eight possible GLC cameras. * It characterizes the distortions seen in reflections according to the local GLC model and collineation and it connects specific distortions to the mirror geometry. * It shows the close relationships between GLCs and the caustic surfaces of rays. In particular, the caustic surface, if it exists, corresponds to the loci of the local cross-slit GLCs. * Finally, I show how to use the GLC analysis to assist mirror design and predict distortions seen in catadioptric imaging systems. 1.4 Thesis Overview This thesis is organized as follows. Chapter 2 describes related work on pinhole and multiperspective camera models and their applications in computer graphics and computer vision. I also highlight the limitations of the current approaches in modeling multiperspective imaging systems. Chapter 3 introduces the General Linear Camera (GLC) model and shows the GLC model describes all pinhole, orthographic, pushbroom, cross-slit, and Epipolar (EPI) cameras, as well as three new camera models. It also includes the derivation of the GLC characteristic equation, which is useful for identifying the 8 atomic GLC types, and proves that this set is complete. Chapter 4 describes algebraic properties of the GLCs. I show GLCs are 2D linear subspaces in the 4D ray space and use the GLC model to analyze higher-order ray subspaces. In addition, I compare GLCs under two-plane-parametrization and under the Plucker coordinates. Chapter 5 develops the a model of GLC projection and collineation. Specifically, the projection maps the 3D scene geometries to the rays and the collineation then 23 maps the rays onto an image plane. I use the projection and collineation theory to explain multiperspective distortions for specific types of the GLCs. Chapter 6 studies arbitrary multiperspective imaging systems by mapping them to ray manifolds. I show how to locally model each multiperspective ray manifold using the GLC model. Furthermore, I derive the Differential Invariant Theorem to show that the local GLC analysis is invariant to the choice of the parametrization planes. Chapter 7 uses the GLC framework for multiperspective rendering. sets of GLCs, when constrained I show that by an appropriate set of rules, can be laid out to tile the image plane to form a piecewise continuous ray manifold, and, hence, generate arbitrary multiperspective renderings. I demonstrate applications of the proposed framework in multiperspective panoramas, neocubist style renderings, and faux-animations. Chapter 8 uses the GLC framework to analyze reflections. I first map the reflected rays off of the mirror to a ray manifold and then use local GLC analysis to characterize the resulting images. GLCs are used to characterize the image distortions seen on arbitrary mirrored surfaces as well as the conventional catadioptric mirrors. Furthermore I show the connection between the GLC model and caustic surfaces of reflections. In particular, I show the "slits" of local GLCs reveal important surface rulings on the caustic surfaces of reflection. I demonstrate applications of the GLC caustic model to improve mirror design, to predict distortions, and to characterize surface properties. Chapter 9 describes future directions for this work and concludes the thesis. 24 Chapter 2 Background and Previous Work This chapter discusses the background and the previous work on multiperspective cameras and images. I first review the classical pinhole (perspective) camera model and its physical setup, imaging process, and limitations. I then discuss alternative camera models known as the multiperspective cameras, including pushbroom, cross-slit, and oblique cameras, as well as real multiperspective catadioptric imaging systems. Finally, I introduce the light-field ray parametrization and review previous work on ray space analysis. 2.1 Cameras The concept of a pinhole camera predates modern history. In particular the model of a camera obscura has been described throughout history and across many cultures. Precise mathematical models of pinhole cameras are fundamental to the fields of photogrammetry, computer vision, and 3D computer graphics. Geometrically, a pinhole camera collect rays passing through a common 3D point, which is called the Center of Projection (COP). Each pinhole camera, therefore, can be uniquely defined by only 3 parameters (the position of COP in 3D). The image of a pinhole camera requires specifying an image plane for projecting these rays. The image transformation due to the change of the image plane is referred to as the homography. 25 The pinhole imaging process, thus, can be decomposed into two parts: projecting the scene geometry into rays and projecting the rays onto the image plane. We refer to the first part as projection and the second as collineation. It has been shown that the projection and collineation can be combined by the classic 3 x 4 pinhole camera matrix [15],which combines six extrinsic and five intrinsic camera parameters into single operator that maps homogenous 3D points to a 2D image plane. These mappings are unique down to a scale factor, and the same infrastructure can also be adapted to describe orthographic cameras. Since pinhole cameras capture similar images to those we observe from our eyes, i.e., in human perspectives, pinhole cameras are also called perspective cameras. The simple pinhole projection model has many nice properties that are useful to computer graphics and computer vision applications. For instance, under the perspective projection, all lines in the scene are projected to lines on the image. Similarly, a triangle in 3D space is projected in a triangle on the pinhole image. Thus, by representing the scene geometry using triangles, one can efficiently render the entire scene by projecting the triangles onto the image plane and then rasterizing the triangles in the image space. Furthermore, in light of two-eye human perception, modern computer vision systems use two pinhole cameras to generate binocular stereo. The perspective projection model also induces the so-called epipolar constraints, which significantly reduces the search space for establishing the correspondences that determine a point's depth. More general camera models do not need to satisfy the same physical constraints that a perspective camera does, i.e., not all rays collected by the camera need to pass through a common point. Such cameras are often referred to as multiperspective cameras and the corresponding images are called multiperspective images. However, most multiperspective cameras are still expected to satisfy some properties in order to capture meaningful (interpretable) images. A list of these properties include: * 2D Ray Subspace. A multiperspective camera, like any camera, collects an image. Typical images, which will be the only one considered in this thesis, are 2-D dimensional subset of the radiant energy along rays. 26 (a) (b) (c) Figure 2-1: Multiperspective images in art. (a) Hall City by M.C. Escher. (b) Portrait of a Young Girl by Pablo Picasso. (c) Red Fish by Henri Matisse. * Continuity. A multiperspective camera should collect a smoothly varying set of rays. This guarantees that continuous objects will project to continuous images. If we further assume all rays can be embedded as points in some rayspace, then the first and second criteria indicate that a multiperspective camera captures rays lying on a 2D continuous manifold in this ray space. * Unique Projection. Excepting singularities (which are discussed in detail later), a multiperspective camera should generally image each 3D point at most once. In a perspective camera, only the point lying at the Center Of Projection (COP) will be imaged multiple times. Likewise, each point in 3D should be imagined no more than once in a multiperspective camera. 2.2 Multiperspective Images In a perspective image, objects in a scene appear as if they would appear from a single viewpoint. Artists have long found this single perspective description too restrictive for illustrating an entire scene. Ideally, an image should be able to smoothly combine different perspectives into a single multiperspective image. Many artists use a multiperspective representation to improve the comprehensibility of the scene and better illustrate the structure and the spatial relationships between the scene objects. Classic examples include the visual paradoxes of Escher, and the Cubism of Picasso and Matisse, as shown in Figure 2-1. Escher uses highly curved projection model to generate "impossible" perspectives of a scene shown in Figure 2-1(a). Picasso and other Cubism pioneers made effective of of rearranging different parts of the depicted scene while maintaining their local spatial relationships, which results in an incongruous spatial systems. Despite the large disparities between projections, the overall impression of a three-dimensional space remains intact, as shown in Figure 2-1(b). Matisse used similar techniques with smoother transitions between different perspectives. In his work Red Fish shown in Figure 2-1(c), the perspective smoothly transitions from horizontal (at the glass container) to the vertical (at the table). Multiperspective images have also been used as backdrops in cel animation to effectively depict camera motion in a single panorama. In cel-animations, a moving foreground element would translate across the background image with no change in parallax. A local camera is attached with the moving foreground element to generate an illusion of motion. In contrast to the foreground elements, backdrop artists often draw by hand a multiperspective projection background to create a deeper, more dynamic environment. Computer generated multiperspective panoramas, as presented by Wood et al [39], combined elements of multiple pinhole cameras into a single image using a semi-automatic image registration process. They relied on optimization techniques, as well as optical flow and blending transitions between views. The concentric mosaics of [32] and [27] are another type of multiperspective image that is useful for exploring captured environments. Durand [9] suggests that multiperspective projections can also be an interactive process and uses them as an example to distinguish between picture generation and user interaction. Agrawala et al [1] developed a system for rendering multiperspective images from three-dimensional models based on spatially varying projections. They point out that the most difficult part of multiperspective rendering using spatially varying projections is resolving occlusion. This difficulty can be relaxed if the transition between projections vary smoothly. Hanson and Wernert [14]addressed the occlusion problem by densely interpolating the path of the ray. The MCOP (multiple center of projection) images of Rademacher [28]are another 28 example of unstructured multiperspective images. They are closely related to images generated pushbroom cameras, but they are not constrained to follow linear paths. While these images were intended as scene representations, they are also interesting and informative images on their own. Glassner [10]described a camera construction for collecting rays from a more general camera model.. In his approach, rays are specified by the two surfaces with common parameterizations. However, none of the previous approaches maintain both the continuity and unique projection property. In general, a desirable multiperspective image design toolbox should guarantee both continuity and uniqueness properties, as well as provide intuitive user controls. Although most of the previous approaches are able to synthesize plausible multiperspective images, very little work has been done on characterizing multiperspective distortions. This is mainly because most of these approaches lack sufficient descriptive power of the local imaging model since, at an infinitesimal level they treat local neighborhoods of rays as pinhole camera approximations. Zorin and Barr show how to correct perceptual distortions in photographs by reprojecting them. Swaminathan and Nayar [35] also analyzed the distortion of multiperspective images with known scene geometry. However, qualitative and quantitative characterizations of multiperspective distortions are missing, resulting in many questions such as what causes multiperspective distortions, what are all possible types of distortions, to what extend one can reduce distortions by manipulating the images, and etc. 2.3 Multiperspective Cameras One way of analyzing a multiperspective image is to consider the hypothetical camera that might have generated it. I will refer to such cameras, whether realizable or not, as multiperspective cameras. Multiperspective camera models have been explored in the computer vision community. These multiperspective cameras include pushbroom cameras [13], which collect rays along parallel planes from points swept along a linear trajectory, two-slit cameras [26], which collect all rays passing through two lines, and oblique cameras, in which every two rays are oblique (neither parallel to 29 (b) (c) Figure 2-2: Pushbroom and Cross-slit Camera. (a) Acquisition geometry of a pushbroom camera. (b) The stationary column sampling routine for synthesizing a pushbroom camera. (c) Acquisition geometry of a cross-slit camera. (d) The linearly varying column sampling routine for synthesizing a cross-slit camera. intersect). Seitz [30] has analyzed the space of stereo cameras to determine those with a consistent epipolar geometry and suggested that pushbroom, cross-slit and linear oblique cameras can all be used to analyze three-dimensional structure, just as pinhole cameras are commonly used. 2.3.1 Commonly Used Multiperspective Cameras The pushbroom camera model was first introduced by Bolles et al [4] for generating Epipolar images (EPI). A pushbroom camera consists of an optical system projecting an image onto a linear array of sensors. The pushbroom sensor is mounted on a moving rail, and as the platform moves, the view plane sweeps out a volume of space and forms a pushbroom image on the sensor as shown in Figure 2-2(a). In practice, the pushbroom view plane is often replaced by a perspective camera and the pushbroom image is synthesized by assembling the same column of each perspective image shown in Figure 2-2(b). Gupta and Hartley [13] investigated the theoretical insights such as the projection and fundamental matrices as well as the epipolar geometry of linear pushbroom cameras. They showed, although the epipolar constraints are totally different from that of a perspective camera, a matrix analogous to the fundamental matrix of perspective cameras exists for pushbroom cameras. Another popular class of multiperspective cameras are the cross-slit cameras. A cross-slit camera has two slits 11 and 12 that are oblique (neither parallel nor coplanar)in 3D space. The camera collects rays that simultaneously pass through the two slits and projects them onto an image plane, as shown in Figure 2-2(c). Zomet et Figure 2-3: Multiperspective images in graphics and vision. (a) and (b) Images captured from catadioptric mirrors (Courtesy of Tomas Pajdla). (c) A reflection image observed on the car window. (d)A pushbroom panorama by University of Washington (Courtesy of Steve Seitz). al [47] carried out an extensive analysis and modelling of cross-slit cameras. They have shown a cross-slit camera can be synthesized similar to the pushbroom camera by stitching linearly varying columns across the pinhole viewing cameras, as shown in Figure 2-2(d). They also showed that the projection and collineation of a cross-slit camera is no longer a 3x4 projection matrix but a 3x4x4 quadratic tensor. Most recently, Pajdla [25] proposed the oblique camera model. The oblique camera is the opposite extremal of the pinhole camera. A pinhole camera collects rays passing through a common point in 3D whereas an oblique camera collects pairwise oblique rays. This means no two rays in an oblique camera can intersect or be parallel. In order to give a complete description of the rays collected by the oblique camera, Pajdla used transition closure to find all rays. He further used the quadratic ruling surface to determine its epipolar geometry. 2.3.2 Catadioptric Mirrors Another class of "real" multiperspective cameras are catadioptric imaging systems that combines a standard perspective camera (which I will refer to as the viewing camera) pointing at one or more mirrored surfaces. An important subclass of catadioptric mirrors places the viewing camera at the geometric focus of a hyperbolic or parabolic surface to synthesize a perspective image with a wider field of view [22, 2] as shown in Figure 2-3(a) and 2-3(b). This setup requires accurate alignment of the viewing camera [18, 36]. When the camera moves out of the alignment, the rays quickly deviate from the desired COP, forming a multiperspective image, and the corresponding imaging geometry is difficult to analyze [37]. Most previous research has been restricted to simple parametric reflectors such as spherical or conical mirrors [8, 40], and equiangular mirrors [34]. Reflection multiperspective images are observed almost everywhere in our daily lives, for example the reflections off of the windows, as shown in Figure 2-3(c). Many of these reflected images exhibit similar distortions as shown in the pushbroom and cross-slit cameras. However, it is unclear where and precisely how they are similar and whether they do correspond to any particular multiperspective camera model or not. Thus, it is desirable to have a multiperspective camera model that applies to reflections off of any parametric or mesh surface and to catalog the expected imaging properties (distortions and ray foci). 2.4 Ray Representation The key to modelling multiperspective cameras is to specify the sets of rays that they collect. The image captured by the camera is then determined by the radiance along each ray that the camera collects. Thus, it is crucial to have an efficient and precise description of the rays. In general, the radiance may vary along a ray in space in presence of participating media and occlusions. To simplify the analysis, I assume that there is no participating media or occlusions within the space I analyze the rays so that these rays can be equivalently represented as lines. 32 Camera Array .. s --- I .s.. Virtual Camera Figure 2-4: An camera array samples a light field with an array of images. A naive approach to represent each line in 3D space is to use a source point and its direction. The source point can be any point (3D) lying on the line and the direction of the line can be its angular orientation (2D), which gives a 5D representation for each line. Although the 5D point-direction representation can describe all possible rays, it is quite redundant because one can choose a different point along the line to describe the same ray. 2.4.1 Light Fields Parametrization An efficient 4D ray representation is the two-plane or light-field parametrization. Light fields are an image-based computer graphics representation that uses a collection of rays as a scene description in place of geometry. In a light-field parametrization, two parallel planes are given in prior and each ray is parameterized using its intersection points with the two planes. Since each intersection point is constrained on a plane, it is 2D, and the over parametrization of the ray is 4D. In theory, light fields capture all the necessary rays within a certain sub-space that are not parallel to the image plane so that a wide range of views within a region can be synthesized, as shown in Figure 2-4. The two parallel planes are often referred to as the st camera plane and uv image plane. In an actual light field setup, an array of images are captured on a regularly sampled 2D plane, as shown in Figure 2-5. Each pixel on an image corresponds to Figure 2-5: Two plane parametrization. A new pinhole image can be synthesized by querying and blending rays from the light field. Each ray is parameterized by intersecting the two parallel planes as [s, t] and [u, v]. a ray that is indexed as [s, t, u, v], where [s, t] and [u, v] is the intersection of the ray with the st plane z = zt and the uv plane z = zu,. Except for rays that are parallel to the uv plane, all rays can be uniquely represented as a 4-tuple [s, t, u, v]. The nice property of 2PP is that rays parameterized under 2PP form a four dimensional vector space. A linear combination of the Is,t, u, v] coordinate of any two rays is still a valid representation of some ray. This contrasts with other parameterizations, such as the 6D Pliickercoordinates [33, 12] which do not form closed vector space. I will give a detailed comparison between 2PP and Pliicker Coordinate in Chapter 4. In this thesis, I use 2PP to represent rays. Unless otherwise specified, I assume Ilim,, to lie on z = 0 plane and its origin to coincide with the origin of the coordinate system and the second plane HIt is at z = 1. I will use interchangeably HII,,a 2.4.2 and Ray Space Analysis As mentioned in the beginning of this chapter, a multiperspective image captures a two dimensional manifold of rays. Thus, under 2PP, all possible multiperspective images correspond to 2D continuous manifolds under this mapping. Of all these 2D manifolds, it is particularly interesting to study the planar manifolds, and many previous work have shown that pinhole cameras are 2D planes in the ray space. Gu et al [12] explored of the linear structures of the rays under two plane parametrization. Their initial results form the foundation of this thesis. Their primary focus was on studying the duality of points and planes under this mapping for characterizing D, 2D, and 3D linear subspaces. In particular, they identified two classes of 2D linear manifolds: cross-slit and pinhole cameras. Pushbroom camera, in their characterization, is a special subclass of the cross-slit when one of its two slits lies at infinity. In addition, they have shown that all rays passing through a line are subject an interesting linear constraint. However, they did not show if there exist other 2D subspaces other than cross-slit and pinholes. My thesis focuses on identifying all possible 2D subspaces and using them to characterize arbitrary multiperspective ray manifolds/cameras. In particular, I show in Chapter 3 that, besides cross-slit, pushbroom, and pinhole cameras, there exist several important but unexplored subsets of multiperspective cameras that also correspond to 2D linear subspaces of rays. I show these cameras can be uniquely characterized using the characteristic equations. Furthermore, [12]only cares about the set of rays lie on a 2D subspace while my work also study the distribution of these rays on the image plane via a general theory of projection and collineation that will be described in Chapter 5. Finally, I show how to use this complete set of 2D ray subspaces for characterize arbitrary ray manifolds for real imaging systems such as catadioptric mirrors. In summary, I have discussed the background and previous work on multiperspective cameras and images in this chapter. I have shown the limitations of the perspective camera model and described many previously discovered multiperspective cameras. I have also introduced the light-field ray parametrization and reviewed the initial work for analyzing the ray space. Before proceeding, I explain my notation. Superscripts, such as pX, py, and pZ represent the x and y and z component of a point or vector. Subscripts, such as fx and fy represents the first-order partial derivatives of f with respect to x and y. And 35 similarly, fx refers to the second-order partial derivative of f with respect to x twice. 36 Chapter 3 General Linear Cameras In this chapter, I introduce the General Linear Camera (GLC) model that is able to describe many common imaging systems. In fact, under an appropriate interpretation, GLCs describe all possible linear images. This provides a single camera model that unifies existing perspective and multiperspective cameras. I also derive characteristic equations that uniquely categorize each variety of GLC. 3.1 General Linear Camera Model The General Linear Camera (GLC) is defined by three rays that originate from three points p(ul, l1), p 2 (u 2 ,v 2 ) and p3 (u 3 ,v 3 ) on an image plane Himage,as is shown in Figure 3-1(a). A GLC collects radiance measurements along all possible "affine combinations" of these three rays. In order to define this affine combination of rays, we assume a specific ray parametrization. W.o.l.g, we define Himageto lie on z = 0 plane and its origin to coincide with the origin of the coordinate system. From now on, we refer to to parameterize rays, we place a second plane ist image as IIuv. In order at z = 1. All rays not parallel to HIt, Iu, will intersect the two planes at s, t, 1] and [u, v, 01respectively. That gives a 4D parametrization of each ray in form [s, t, u, v]. This ray representation, called the two-plane parametrization (2PP), is widely used by the computer graphics community for representing light fields and lumigraphs [21, 11]. Under this parametrization, 37 an a'r (a) Figure 3-1: General Linear Camera Model. a)A GLC is characterized by three rays originated from the image plane. b)It collects all possible affine combination of three rays. affine combination of three rays ri[si, ti, ui, vi], i = 1, 2, 3, is defined as: r = a " [sl, tl, Ui, V1] + ) * [S2 ,t 2 , U2 ,v 2 ] + (1- -- ) . [S3, t3, U3, V3] (3.1) The choice of Hst at z = 1, is, of course, arbitrary. One can choose any plane parallel to HII, to derive an equivalent parametrization. Moreover, these alternate parameterizations will preserve affine combinations of three rays. Theorem 3.1-1. The affine combinations of any three rays under two different 2PP parameterizationsthat differ by choice of IIt (i.e., [s, t, u, v] and [s', t',u, v] ) are the same. Proof: Suppose IHI,, is at some arbitrary depth zo, zo # 0. Consider the transformation of a ray between the default parametrization (zo = 1) and this new one. If r [s, t, u, v] and r[s', t', u, v] represent the same ray r in 3D, then r[s, t, u, v] must pass through [s', t', zo], and there must exist some A such that A. [s, t, 1] + (1 - A) - [u, v, 0] = [s', t', zo] Solving for A, we have s' =s.zo+u.(1-zo), t'=tzo+v.(1-Zo) (3.2) Since this transformation is linear, and affine combinations are preserved under linear transformation, the affine combinations of rays under the default two-plane parametrization (zo = 1) will be consistent for parameterizations over alternative parallel planes. Moreover, the affine weights for a particular choice of parallel lit are general. We call the GLC model "linear" because it defines all 2-dimensional affine subspaces in the 4-dimensional "ray space" imposed by a two-plane parametrization. Moreover, these 2D affine subspaces of rays can be considered as images. We refer to the three rays used in a particular GLC as the GLC's generator rays. Equivalently, a GLC can be described by the coordinates of two triangles with corresponding vertices, one located on list, and the second on II. Unless otherwise specified, we will assume the three generator rays (in their 4D parametrization) are linearly independent. This affine combination of generator rays also preserves linearity, while other parameterizations, such as the 6D Pluicker coordinates [33], do not [12]. Theorem 3.1-2. If three rays are parallel to a plane II in 3D, then all aine combinations of them are parallel to II as well. Proof: Let r[sl, l, v], r 2 [s2 , t 2 , u 2 ,v 2], and r 3 [S3, t 3 , U3, V3] be the three generator rays that are parallel to plane II. Let h be the unit normal of plane II. Since rl, r 2 , and r3 are all parallel to II and ri has direction di = [si - ui, ti - vi, 1], we must have = The direction of a ray that is the affine combination of the three generator rays is daffine = (di + d2 + (1 - ac- /3)d3. By distributing the inner products, it is clear that it is easy to see that daffine h = 0. Thus, any affine combination of these three rays will be parallel to II as well. Theorem 3.1-3. If three rays intersect a line I parallel to the image plane, all 39 rC In\ Ia (D) kL% Figure 3-2: Classic camera models represented as GLC. (a)Two similar triangles on two planes define a pinhole camera; (b)Two parallel congruent triangles define an orthographic camera; (c)Three rays from an cross-slit camera. affine combinations of them will intersect 1 as well. Proof: By theorem 3.1-1, we can reparametrize three rays by placing Hst so that it contains 1 without changing the affine combinations of the three rays. Because the st plane intersections of the three rays will lie on 1, all affine combinations of three rays will have their st coordinates on 1, i.e., they will all pass through 1. The same argument also holds when all rays which pass through a given point. 3.2 Equivalence of Classic Camera Models Traditional camera models have equivalent GLC representations. Pinhole camera: By definition, all rays of a pinhole camera pass through a single point, C in 3D space (the center of projection). Any three linearly independent rays from C will the intersect II,, and IIst planes to form two triangles. These triangles will be similar and have parallel corresponding edges, as shown in Figure 3-2(a). Furthermore, any other ray, r, through C will intersect II., and Ht planes at points V>,, and 4qt. These points will have the same affine coordinates relative to the triangle vertices on their corresponding planes, and r has the same affine coordinates as these two points. Orthographic camera: By definition, all rays on an orthographic camera have the same direction. Any three linearly independent rays from an orthographic camera intersect parallel planes at the vertices of congruent triangles with parallel corresponding edges, as shown in Figure 3-2(b). Rays connecting the same affine combination of these triangle vertices, have the same direction as the 3 generator rays, and will, therefore, originate from the same orthographic camera. Pushbroom camera: A pushbroom camera sweeps parallel planes along a line I collecting those rays that pass through 1. We refer to this family of parallel planes as II*. We choose IIv, parallel to but not containing 1, and select a non-degenerate set of generator rays (they intersect IHuvin a triangle). By Theorem 3.1-2 and 3.13, all affinriecombinations of the three rays must all lie on II* parallel planes and must also pass through and, hence, must belong to the pushbroom camera. In the other direction, for any point P on Iuv, there exist one ray that passes through P that intersects I1and is parallel to II*. Since P must be some affine combination of the three vertexes of the uv triangle, the ray must lie on the corresponding GLC. Furthermore, because all rays of the pushbroom camera will intersect Iuv, the GLC must generate equivalent rays. cross-slit camera: By definition, an cross-slit camera collects all rays passing through two non-coplanar lines. We choose IIuv to be parallel to both lines but not containing: either of them. One can then pick a non-degenerate set of generator rays and find their corresponding triangles on lst and Huv.. By Theorem 3.1-3, all affine combinations of these three rays must pass through both lines and hence must belong to the cross-slit camera. In the other direction, authors of cross-slit [26, 47] have shown that each point on the image plane Iu,, maps to a unique ray, r, in an cross-slit camera. Since p must be some affine combination of the three vertexes of the uv triangle, r must belong to the GLC. The GLC hence must generate equivalent rays as the cross-slit camera. Epipolar Plane Image: EPI [4] cameras collect all rays that lie on a plane in 3D space. We therefore can pick any three linearly independent rays on the plane as generator rays. Affine combinations of these rays generate all possible rays on the plane,so long as they are linearly independent. Epipolar Plane Images. 41 Therefore a GLC can also represent 3.3 Characteristic Equation of GLC Although we have shown that a GLC can represent many commonly used camera models, the representation is not unique (i.e., different sets of three generator rays can define the same camera). In this section we develop a criterion to classify general linear cameras. One discriminating characteristic of affine ray combinations is whether or not all rays pass through a line in 3D space. This characteristic is fundamental to the definition of many multi-perspective cameras. We will use this criteria to define the the first characteristic equation of general linear cameras. Recall that any 2D affine subspace in 4D can be defined as affine combinations of three points. Thus, GLC models can be associated with all possible planes in the 4D since GLCs are specified as affine combinations of three rays, whose duals in 4D are the three points. Theorem 3.3-1. Given a non-EPI, non-pinhole GLC, if all camera rays pass through some line 1, not at infinity, in 3D space, then must be parallel to Hu,. Proof: We demonstrate the contrapositive. If is not parallel to Ilu,, and all rays on a GLC pass through , then we show the GLC must be either an EPI or a pinhole camera. Assume the three rays pass through at least two distinct points on , otherwise, they will be on a pinhole camera, by Theorem 3.1-3. If is not parallel, then it must intersect IIst, IIH,, at some point [so, to, 1] and [o, vo, 0]. Gu et al [12] has shown all rays passing through must satisfy the followingbilinear constraint (u-uo)(t- to)-(v-vo)(s -so) =0 (3.3) We show that the only GLCs that satisfy this constraint are EPIs or pinholes. All 2D affine subspaces in s, t, u, v] can be written as the intersection of two linear constraints Ai s + Bi t + Ci u + Di v + Ei = 0, i = 1, 2. (See Chapter 4.2 for details.) In general we can solve these two equations for two variables, for instance, 42 we can solve for u-v as u=A .s+B.t+E , v=A. s+B2 t+E (3.4) Substituting u and v into the bilinear constraint (3.3), we have (A .s + B; t + E- uo)(t - to) = (A' s + B' t + E2- vo)(s- (3.5) This equation can only be satisfied for all s and t if A = B' and B1 = A = 0, therefore, equation (3.4) can be rewritten as u = A' s + E' and v = A'. t + E2. Gu et al [12] have shown all rays in this form must pass through a 3D point P (P cannot be at infinity, otherwise all rays have uniform directions and cannot all pass through any line 1, not at infinity). Therefore all rays must lie on a 3D plane that passes through I and finite P. The only GLC camera in which all rays lie on a 3D plane is an EPI. If the two linear constraints are singular in u and v, we can solve for s-t, and similar results hold. If the two linear constraints cannot be solved for u-v or s-t but can be solved for u-s or v-t, then a similar analysis results in equations of two parallel lines, one on rIst, the other on Iluv. The set of rays through two parallel lines must lie on an EPI. Theorem 3.1-3 and 3.3-1 imply that given a GLC, we need only consider if the three generator rays pass through some line parallel to rIst. We will show an alternative proof of Theorem 3.3-1 using Pliucker Coordinate in Chapter 4. We use this relationship to define the characteristic equation of a GLC. The three generator rays in a GLC correspond to the following 3D lines: ri = Ai [si,ti, 1]+ (1-A i) [ui, vi, ] i = 1,2,3 The three rays intersect some plane HI= parallel to Hu, when A1 = A2 = A3 = A. By Theorem 3.1-3, all rays on the GLC pass through some line 1 on HI,=, if the three generator rays intersect 1. Therefore, we only need to test if there exist any A so that the three intersection points of the generator rays with II,=, lie on a line. A necessary 43 and sufficient condition for 3 points on a constant z-plane to be co-linear is that they have zero area on that plane. This area is computed as follows (Note: the value of z is unnecessary): (A Si1+ (1 - A)* Ul) (A. 1 .v 2) 1 V3 ) 1 (A.t .u 2 ) 82+(1-A) (A/ t I + (1 - A)* V1 ) (A. s 3 + (1 -A) u 3 ) 2 +(1-A) (A. t 3 + (1 -A) =0 (3.6) Notice equation (3.6) is a quadratic equation in A of the form A A2 + B A+ C = 0 (3.7) where A= B= S1 - U1 t s 2 -u 2 t2 -v s 3 -u 3 t3 -v S1 V1 1 S2 v2 1 s3 V3 - t 3 2 1 , C= u 2 v2 1 , 3 1 U3 sV3 1 U 3 1 U1 t2 - V1 1 V1 2 U 1 1 -2 (3.8) 1 U1 V1 1 U2 V2 1 U3 3 1 I call equation 3.9 the first characteristic equation of a GLC. Since the characteristic equation can be calculated from any three rays, one can also evaluate the characteristic equation for EPI and pinhole cameras. The number of solutions of the characteristic equation implies the number of lines that all rays on a GLC pass through. It may have 0, 1, 2 or infinite solutions. The number of solutions depends on the denominator A and the quadratic discriminant A = B 2 - 4AC. Notice, a GLC can also be equivalently generated by two generator triangles on the st and uv plane. The followingTheorem shows the translation invariant property of a GLC. Theorem 3.3-2. Given a GLC, three generator rays [si,ti, ui, vi], i = 1, 2, 3, 44 translating the two generator triangles [s', ti] = [si + T, ti + Tt], [u', v'] = [ui+ Tu, vi + Tv], i = 1, 2, 3, do not change the coefficients A, B and C of the characteristic equation of the GLC. Proof: This is because the determinants used in the expressions of A, B and C are invariant to translation. For instance, s1-u 1 t1 -v 1 1 +Ts- U -u T S2 -u11 2 t2 - 2 1 = S2+ T - 2 -Tu S3 -'u11 3 t3 - 3 1 s3 +Ts-u 3 -Tu t +Tt-vi -Tv 1 t 2 + Tt - v 2 t 3 +Tv-V Tv 1 =A 3 -Tv (3.9) 1 Similar results hold for B and C. 3.4 Characterizing Classic Camera Models In this section, I show how to identify standard camera models using the characteristic equation of 3 given generator rays. Theorem 3.4-1. Given a GLC, three generator rays, and its characteristic equation A A2 + B A + C = 0, then all rays are parallel to some plane if and only if A=0. Proof: Notice in the matrix used to calculate A, row i is the direction di of ray ri. Therefore A can be rewritten as A = (d1 x d 2 ). d3 . Hence A = 0 if and only if d-, d 2 and d3 are parallel to some 3D plane. And by Theorem 3.1-2, all affine combinations of these rays must also be parallel to that plane if A = 0. 3.4.1 A = 0 case When A = 0, the characteristic equation degenerates to a linear equation, which can have 1, 0, or an infinite number of solutions. By Theorem 3.4-1, all rays are parallel to some plane. Only three standard camera models satisfy this condition: pushbroom, orthographic, and EPI. All rays of a pushbroom lie on parallel planes and pass through one line, as is 45 shown in Figure 3-4(a). A GLC is a pushbroom camera if and only if A = 0 and the characteristic equation has 1 solution. All rays of an orthographic camera have the same direction and do not all simultaneously pass through any line 1. Hence its characteristic equation has no solution. The zero solution criteria alone, however, is insufficient to determine if a GLC is orthographic. We show in the following section that one can twist an orthographic camera into bilinear sheets by rotating rays on parallel planes, as is shown in Figure 3-4(b), and still maintain that all rays do not pass through a common line. In Section 3, we showed that corresponding edges of the two congruent triangles of an orthographic GLC must be parallel. This parallelism is captured by the following expression: (si-sj) (ui-uj) i j = 1,2,3 and i j (ti-tj) (vi-vj) (3.10) We call this condition the edge-parallelcondition, and define it as the second characteristic equation of a GLC.. It is easy to verify that a GLC is orthographic if and only if A = 0 (all rays are parallel to at least one plane), its characteristic equation has no solution, and it satisfies the edge-parallelcondition. Rays of an EPI camera all lie on a plane and pass through an infinite number of lines on the plane. In order for a characteristic equation to have infinite number of solutions when A = 0, we must also have B = 0 and C = 0. This is not surprising, because the intersection of the epipolar plane with Hst and lHuvmust be two parallel lines and any set of three generator rays must satisfy A = 0 (because they lie on a plane), C = 0 (because the area formed on the three rays on uv plane is zero), and B=0. 3.4.2 A $ 0 case When A $ 0, the characteristic equation is quadratic and can have 0, 1, or 2 solutions depending on the characteristic equation's discriminant A. We show how to identify the remaining two classical cameras, pinhole and cross-slit cameras in term of A and 46 A. All rays in a pinhole camera pass through the center of projection (COP). Therefore, any three rays from a pinhole camera, if linearly independent, cannot all be parallel to any plane, and by Theorem 3.4-1, A # 0. Notice that the roots of the characteristic equation correspond to the depth of the line that all rays pass through, hence the characteristic equation of a pinhole camera can only have one solution that corresponds to the depth of the COP, even though there exists an infinite number of lines passing through the COP. Therefore, the characteristic equation of a pinhole camera must satisfy A $ 0 and A = 0. However, this condition alone is insufficient to determine if a GLC is pinhole. In the following section, we show that there exists a camera where all rays lie on pencil of planes sharing a line, as shown in Figure 3-4(c), which also satisfies these conditions. One can, however, use the second GLC characteristic equation (the edge -parallel condition) to verify if a GLC is pinhole. Thus a GLC is pinhole, if and only if A 0, it first characteristic equation has one solution, and it satisfies edge-parallelcondition. Rays of an cross-slit camera pass through two slits and, therefore, the characteristic equation of a GLC must have at least two distinct solutions. Furthermore, Pajdla [26] has shown all rays of an cross-slit camera cannot pass through lines other than its two slits, therefore, the characteristic equation of an cross-slit camera has exactly two distinct solutions. Thus, a GLC is an cross-slit if and only if A $ 0 and A > 0. 3.5 New Multiperspective Camera Models The characteristic equation also suggests three new multiperspective camera types that have not been previously discussed. They include 1)twisted orthographic: A = 0, its first characteristic equation has no solution, and all rays do not have uniform direction; 2)pencil camera: A # Oand the equation has one root, but all rays do not pass through a 3D point; 3)bilinear camera: A $ 0 and the characteristic equation has no solution. In this 47 I z' f3 ,•-- . /r 2 /l r2 4. (a) - - - ,r - I - I r Ir F1. N1 I -I (C) - Figure 3-3: Bilinear Surfaces. (a) r3 is parallel to S; (b) r 3 is parallel to S, but still intersects S; (c) r3 is not parallel to S, and does not intersect S either. section, we give a geometric interpretation of these three new camera models. Before describing each new camera model, however, we will first discuss a helpful interpretation of the spatial relationships between the three generator rays. An affine combination of two 4D points defines a 1-dimensional affine subspace. Under 2PP, a 1-D affine subspaces corresponds to a bilinear surface S in 3D that contains the two rays associated with each 4D point. If these two rays intersect or have the same direction in 3D space, S degenerates to a plane. Next, we consider the relationship between ray Tr 3 and S. We define r3 to be parallel to S if and only if r 3 has the same direction as some ray r E S. This definition of parallelism is quite different from conventional definitions. In particular, if r3 is parallel to S, r3 can still intersect S. And if r3 is not parallel to S, r3 still might not intersect S, Figure 3-3(b) and (c) show examples of each case. This definition of parallelism, however, is closely related to A in the characteristic equation. If r3 is parallel to S, by definition, the direction of r3 must be some linear combination of the directions of rl and r2 , and, therefore, A = 0 by Theorem 3.4-1. A = 0, however, is not sufficient to guarantee r3 is parallel to S. For instance, one can pick two rays with uniform directions so that A = 0, yet still have the freedom to pick a third so that it is not parallel to the plane, as is shown in Figure 3-3(c). The number of solutions to the characteristic equation is also closely related to the number of intersections of r3 with S. If r3 intersects the bilinear surface S(rl, r2 ) at P, then there exists a line 1, where P E 1, that all rays pass through. This is -L~PGt~-·="(a) (b) (c) Figure 3-4: Pushbroom, Twisted Orthographic, and Pencil Cameras. (a) A pushbroom camera collects rays on a set of parallel planes passing through a line; (b) A twisted orthographic camera collects rays with uniform directions on a set of parallel planes; (c) A pencil camera collects rays on a set of non-parallel planes that share a line. because one can place a constant-z plane that passes through P and intersects r, and r2 at Q and R. Because S is a bilinear surface, the cut by a constant-z plane on S is a line. Therefore, P, Q and R must lie on a line and all rays must pass through line PQR. Hence r 3 intersecting S(rl, r2 ) is a sufficient condition to ensure that all rays pass through some line. It further implies that, if the characteristic equation of a GLC has no solution, no two rays in the camera intersect. GLCs whose characteristic equation has no solution are examples of oblique cameras from [25]. 3.5.1 New Camera Models Our GLC model and its characteristic equations suggests 3 new camera types that have not been previously described. Twisted Orthographic Camera: The characteristic equation of the twisted orthographic camera satisfies A = B = 0 and C 5 0, thus, has no solution, and its generators do not satisfy the edge-parallel condition. If rl, r2 and r3 are linearly independent, no solution implies r3 will not intersect the bilinear surface S. In fact, no two rays intersect in 3D space. In addition, A = 0 also implies that all rays are parallel to some plane II in 3D space, therefore the rays on each of these parallel planes must have uniform directions as is shown in Figure 3-4(b). Therefore, twisted orthographic cameras can be viewed as twisting the parallel planes of rays in an orthographic camera along common bilinear sheets. Table 3.1: Characterize General Linear Cameras by Characteristic Equation Characteristic Equation 2 Solution 1 Solution A 0 A= 0 0 Solution Inf. Solution cross-slit Pencil/Pinholet Bilinear 0 0 Pushbroom Twisted/Ortho.t EPI t: A GLC satisfying edge-parallel condition is pinhole(A y! 0) or orthographic (A = 0). Pencil Camera: The characteristic equation of a pencil camera satisfies A # 0, has one solution and the generators do not satisfy the edge-parallelcondition. In Figure 3-4(c), we illustrate a sample pencil camera: rays lie on a pencil of planes that share line 1. In a pushbroom camera, all rays also pass through a single line. However, pushbroom cameras collect rays along planes transverse to planes of a pencil camera contains whereas the (i.e., lie in the pencil of planes through ), as is shown in Figure 3-4(a) and 4(c). Bilinear Camera: By definition, the characteristic equation of a bilinear camera satisfies A $ 0 and the equation has no solution ( < 0). Therefore, similar to twisted orthographic cameras, no two rays intersect in 3D in a bilinear camera. In addition, since A $ 0, no two rays are parallel either. Therefore, any two rays in a bilinear camera form a non-degenerate bilinear surface, as is shown in Figure 3-3(a). The complete classification of cameras is listed in Table 3.1. 3.5.2 All General Linear Cameras Recall that; the characteristic equation of a GLC is invariant to translation, therefore we can translate (sl,tl) to (0,0) to simplify computation. Furthermore, we assume the uv triangle has canonical coordinates (0, 0), (1, 0) and (0, 1). This gives: A = s2 t3 -s 3t2 - 2 -t 3 + 1, A = (s2 - t3)2 + 4s3t2 (3.11) The probability that A = 0 is very small, therefore, pushbroom, orthographic and twisted orthographic cameras are a small subspace of GLCs. Furthermore since s2, t2 , 50 S3 and t3 are independent variables, we can, by integration, determine that approximately two thirds of all possible GLCs are cross-slit, one third are bilinear cameras, and remainders are other types. 3.6 Example GLC Images In Figure 3-5, we compare GLC images of a synthetic scene. The distortions of the curved isolines on the objects illustrate various multi-perspective effects of GLC cameras. In Figure 3-6, we illustrate GLC images from a 4D light field. Each GLC is specified by three generator rays shown in red. By appropriately transforming the rays on the image plane via a 2D omography, most GLCs generate easily interpretable images. In. Figure 3-7, we choose three desired rays from different pinhole images and fuse them into a multiperspective bilinear GLC image. GLC images can be rendered directly by ray tracing a synthetic scene, or by cutting through pre-captured light fields. In Figure 3-8, we synthesize a cross-slit and cross-slit camera by cutting through a 3D light field along different directions. By appropriately organizing rays, both GLCs generate interpretable images similar to pinhole and orthographic cameras. The pencil camera shown on the right of Figure 3-8 is highly twisted. This is because we are moving the viewing angle from left to right while scanning the image from the top to the bottom. Images of different GLC types also exhibit unique distortions. For instance, all lines in 3D space are preserved as lines on the 2D image for orthographic and pinhole camera while they are curved in most other GLCs. In particular, lines are strongly twisted in the images of twisted orthographic and pencil camera. This is mainly because the rays collected by both cameras are twisting in 3D space in nature. Similar to pinhole cameras, many GLCs have "vanishing" points of parallel lines. Furthermore, in pushbroom camera, objects far away from the camera are stretched while their size is preserved under pinhole camera, orthographic, and cross-slit cameras. I will provide a complete characterization of the GLC images in Chapter 5. 51 I; 1.: I-. ~ s-~ .1-,- ';ýP_ -J - ul I/ Figure 3-5: Comparison between synthetic GLC images. From left to right, top row: a pinhole, an orthographic and an EPI; middle row: a pushbroom, a pencil and an twisted orthographic; bottom row: a bilinear and an cross-slit. r Figure 3-6: GLC images created from a light field. Top row: a pencil, bilinear, and pushbroom image. Bottom row: an cross-slit, twisted orthographic, and orthographic image. Figure 3-7: A multiperspective bilinear GLC image synthesized from three pinhole cameras shown on the right. The generator rays are highlighted in red. Figure 3-8: Synthesizing GLCs by cutting through a 3D horizontal light fields. Left: we cut through the light field vertical diagonally to generate a cross-slit GLC. Right: we cut through the light field vertical diagonally to generate a pencil GLC. 54 Chapter 4 GLC Characteristics This chapter explains several important geometric and algebraic properties of the GLC model. I first introduce the relative two-plane-parametrization (R-2PP) to simplify te GLC characteristic equations, and show how it can be interpreted as the area formed by the three generator rays on a sweeping plane. The solutions to the first characteristic equation correspond to slits at both finite and infinite distance. I then derive all possible transitions one type of GLC to another by changing its slit geometry. Next, I explore several algebraic properties of the GLC model. I show GLCs are 2D linear subspaces in the 4D ray space and can be alternatively represented by two linear constraints. The two linear constraints satisfy specific algebraic properties for different types of GLCs. Finally, I compare GLCs defined under two-plane-parametrization and under Pliicker Coordinate. 4.1 Relative Two Plane Parametrization GLCs are defined as the affine subspaces of three generator rays that are parameterized by their intersections with two parallel planes at s, t, 1] and [u, v, 0]. Therefore, the two-plane-parametrization of the ray can be viewed as a two-point parametrization of a line. Alternatively, each ray can be parameterized by a point and a direction. If we choose the point as u, v, 0] and direction as [a, T, 1] = [s,t, 1] - [u, v, 0], we have a point-direction-parametrization of a ray [a, T, u, v]. Since [, T, 1] can also be viewed 55 as the relative coordinate of [s,t, 1] with respect to [u, v, 0], I refer to this parametrization as the relative-two-plane-parametrization (R2PP). The transformation from 2PP to R2PP is linear, and more precisely a shearing transformation, thus, affine subspaces are preserved under R2PP. We can reparameterize a GLC by representing the rays under R2PP as: GLC = a [ 1 , T1,U1 , V] + [0'2, 2 , U 2, V2] + (1- a - ) -[0'3 , T3 , U3 , 3] (4.1) The first characteristic equation 3.8 under R2PP simplifies to ul+ A- al u2 + U3 A + A . Vl +- 2 03 T1 1 V2 + A 2 1 v 3 + A* 3 1 (4.2) =0 resulting in a simplified quadratic equation AA2 + BA + C = 0 where 1r O1 A = B = C= Ul V1 1 U2 V2 1 U3 V3 1 0' 2 T2 0' 3 rT 3 1 O1 V1 1 U1 1 0'2 V2 1 U2 1 0'3 V3 1 1 , (4.3) U3 1 The second characteristic equation 3.12 (edge-parallel condition) under 2PP is then transformed as: (,i-Uj) = (u-uj) (i -j) i,j = 1,2,3 and i j (4.4) (Vi -Vj) From now on, I will use the characteristic equations under R2PP throughout the thesis. 56 k = 0.0 = 2.3 k = 5.2 Figure 4-1: From left to right: the area formed by the three generators at each sweeping plane. Green corresponds to positive areas and red corresponds to negative. When the area is zero, all rays will pass through a slit on the corresponding sweeping plane. 4.2 Sweeping Area Recall from chapter 3.3 that the left hand side of the first characteristic equation of a GLC computes the area formed by the intersections of the three generator rays with each plane at depth A shown in Figure 4-1. Thus, we can plot the area formed the generator rays vs. the depth of the plane: ul +Aal 2. Area(A)= u2 + A o 2 u 3 + A.o3 vl + +A- 1 1 v 2 + A' 2 1 =AA2 + BA+C V3 + A 3 1 (4.5) where A, B, and C can be computed in equation 4.3. The area-depth curve is, in general, quadratic in A with either 0, 1, or 2 intersections with the area = 0 axis. The roots correspond to the depth of the slits(s) that all rays in the GLC pass through. For example, cross-slit cameras should always have two intersections with the area = 0 axis. Pencil and Pinhole cameras have a single intersection and are tangent to the area = 0 axis. The area-depth curve of a bilinear cameras never intersects the area = 0 axis and, therefore, maintains either positive or negative definite areas. Furthermore, the area-depth curve degenerates from a quadratic into a line when A = 0, which implies all rays are parallel to some plane. This happens with pushbroom, orthographic, twisted orthographic, and EPI cameras, as shown in Table (3-1). By analysis in Chapter 3, pushbroom cameras also satisfy B # 0. Therefore, the area-depth curve has a slope. The first characteristic equation of the orthographic and twisted orthographic cameras satisfy A = B = 0, thus, the Pinhole/Pencil Area Ortho/T. Ortho. Area lamda Pushbroom Area lamda Cross-slit Bilinear Area Area mdaamda EPI Area lamda lamda Figure 4-2: Area-Depth Curve (ADC) for GLCs. From left to right: ADC of a pinhole/pencil camera is quadratic and intersects area = 0 axis once; ADC of orthographic/twisted orthographic camera is a constant line parallel to area = 0 axis; ADC of a pushbroom camera is a line that intersects area = 0 axis once; ADC of a cross-slit camera is quadratic and intersects area = 0 axis twice; ADC of a bilinear camera is quadratic and never intersects area = 0 axis; ADC of an EPI camera is the area = 0 axis. area formed the three generator rays will remain constant and the area- depth curves lie parallel to the area axis. Finally, the area-depth of an EPI camera coincides with the area = 0 axis because all rays lie on a plane. The complete characterization is shown in Figure 4-2. Furthermore, since any quadratic curve is uniquely defined by three points, we can determine the first characteristic equation and the type of a GLC if we know the areas formed by the generator rays on three different depth planes. 4.3 GLC Slits The slits of a GLC determine the fundamental behavior of the rays. When the slit changes, the structure of these rays also changes accordingly, and in the extreme, a slit can move to infinity and the GLC characteristics change dramatically. However, the first characteristic equation only allows A with finite values as solutions. Ideally, we would like to have a uniform description of slits at both finite and infinite depths as solutions to the first GLC characteristic equation. Note that sets of the parallel planes intersect at an infinite line. Algebraically, this corresponds to GLCs with a characteristic equation coefficient of A = 0, and by Table 3.1, only pushbroom, twisted orthographic, orthographic, and EPI have infinite slits. In the following, I re-characterize GLC in terms of finite and infinite lines that Table 4.1: Characterize GLC with Lines Number of Lines GLC Type 0 line class 1 line class 2 line class oo line class Bilinear Pencil (1 finite); Twisted Orthographic (1 infinite); XSlit (2 finite); Pushbroom (1 finite, 1 infinite); Pinhole (oo finite); (Orthographic (o infinite); EPI (oo finite and infinite); they intersect: 0-Line Class GLC: Bilinear camera is the only GLC that all rays pass through no finite or infinite lines. 1-Line Class GLCs: Rays in a pencil camera pass through one finite line. Rays in an twisted orthographic camera passes through one infinite line. A twisted orthographic camera is formed by pushing the finite line in the pencil GLC to infinity. When the line in a pencil shrinks to point, it transitions to a pinhole camera. If we push the point to infinity, it becomes an orthographic camera. 2-Line Class GLC Rays in a cross-slit camera pass through two finite lines. When one of the lines goes to infinity, a cross-slit transitions to a pushbroom camera. When both lines in a cross-slit camera go to infinity, it transitions to an orthographic camera. When the two slits intersect, it transitions to a pinhole camera. When the two slits approach to coplanar, it transitions to an EPI image. The complete characterization is shown in Table 4.2. It describes all possible transitions between GLCs in terms of the change in slits. Notice, there is no possible transitions between 0-line, 1-line, and 2-line classes. Most recently, Roman et al 29] used a similar analysis for interactive designing panorama images. We will derive the transitions of the GLC on a differentiable continuous ray manifold and use them for generating various multiperspective rendering effects in Chapter 7. 59 4.4 Linear Ray Manifolds GLCs describe 2D affine subspaces of rays. When parameterized under 2PP, GLCs are 2D hyperplanes in a 4D vector space. Any 2D hyperplane in 4D can also be defined via two linear constraints. In this section I discuss the mappings from a GLC to two linear constraints and study the algebraic properties these linear constraints should satisfy for specific GLCs types. 4.4.1 Canonical GLC Representation Recall that a GLC is defined as an affine combination of three generator rays, each represented as (ai, i, ui, vi], i = 1, 2, 3. Thus, it takes 12 variables to define a GLC. However, this GLC representation is not unique since one can pick a different set of three rays to define the same GLC. To enforce uniqueness, I choose the three generator rays in a specific form as [al, Ti, 0, 01, [ [1, 0, 0], ad 2 , T2 , 1, 0], and [a3 , 3 , 0, 1], i.e., the three rays originate from [0,0, 0], [0, 1, 0] from the image plane. I call this parametrization the canonical GLC representation. Except for those GLCs that have a slit lie on the uv plane, all other GLC's can be uniquely represented using this 6-parameter canonical representation. Since the uv plane corresponds to the image plane, using the canonical GLC representation is similar to specifying the texture coordinate of a triangle, where the texture coordinate here represents the direction [a, r] of the ray. I will use the canonical representation of the GLC to study the multiperspective imaging properties in Chapter 5. 4.4.2 GLC to Linear Constraints In order to find the pair of linear constraints associated with a particular GLC, we represent the GLC in its canonical form as [a, , v] = a [o1 ,r 1 ,0,0] + . [0'2 , -2,1,0] + (1- -a -3). [0'3, 3, 0, 1] 60 (4.6) Solving for a, r, u, and v gives two linear constraints a =' (1 -u-v) T = (1-u + a 2 + a3 = ( 2 -a I)u + ( 3 - 1)v + 1 -V)T1 + UT2 + VT3 = ( 2 - T1)U+ (3- (4.7) I-)V + LT 1 Next, I introduce a vector notation to simplify the analysis, where each linear constraint equation A.+BT+Cu+Dv+E = 0 is represented as the vector [A,B, C, D, E]. As a result, the two constraints of a GLC can be written as: r = [1, , a1 - a 2, a1 -a0'3,-a1 ] r2 = [0,1,T- (4.8) T2 ,7T1 - T3 , -7'1] The two constraints are unique for each GLC and I call them the canonical constraints of a GLC. All rays in a GLC should satisfy its canonical constraints. Furthermore, any linear combination of the two canonical constraints gives another linear constraint that the GLC must also satisfy. We are particularly interested in the special forms that these linear constraints take for each specific GLC type. In a pinhole camera, all rays pass through a 3D point [px,pY, pZ]. Therefore, each ray [a0,r, u, v] of a pinhole GLC must, for some value of A, satisfy: [u,v,0] + A[a, , 1] = [pX,py, pZ] (4.9) It is easy to see A = pZ. Hence the pinhole GLC is subject to the linear constraints u + pZa' = px (4.10) + pZT' = pY or in vector form, 1, 0, l/pZ, 0, _px/pZ] and [0,1, 0, l/pZ, py/pZ]. This means some linear combination of the two canonical constraints (4.8) of the GLC should satisfy 61 the two specific pinhole constraints above, i.e., 3a, 1, a 2 , 02 alrF 1 +/ 1F2= [1,0, l/pZ, 0, -pX/pZ] a 2 r1 +132r2 = [0, 1,0, l/pZ, -pY/pZ] Substituting Equation 4.8 into 4.11, we get al = 2= 1 and (4.11) a2 = 31 = 0. Solving for the remaining terms gives the second characteristic equation (edge-parallel condition) of the GLC (3.12). For those GLC types where every ray is parallel to at least one common plane (Pushbroom, Orthographic, Twisted-Orthographic, and EPI), it is equivalent to say that, these rays must be perpendicular to the plane's normal. Let the normal of the plane be nr= [nX,n y , nZ]. Then we must have [a 7, 1][nx, ny, nz] = 0 (4.12) nx n (4.13) or, yrn+ Z = 0 This implies some linear combination of the two canonical linear constraints of the GLC should give the two specific constraints 4.13, i.e., 3a, /3 al' + /3r2 = [nx , n y , 0, 0, nZ] (4.14) Solving equation 4.14 gives A = 0 of the first canonical GLC characteristic equation 4.2. For orthographic cameras, all rays have a common direction, i.e., a = a0 and T = To. trhus, there exist some linear combinations of the two canonical linear constraints that satisfy the two orthographic constraints, i.e., 3 a1 F1 + /1F2 = a 2 F 1 + "2F2 = 62 [1, 0, 0, 0, -o] [0,10, 0, -o] 1, 1, a 2 , 2 (4.15) Solving for the remaining terms in equation 4.15 gives ai = uo, and i = To, for i = 1, 2, 3. Finally, if all rays in a GLC pass through a slit parallel to the uv plane at z = pZ plane, we can parameterize the line with a point and its direction parallel to the uv plane as [pX,py, pZ] + u[dx,d y , 0]. Each ray [a, r, u, v] will intersect the slit, therefore, there must exist some A, p for each ray such that [u, v, O] + A[a, r, 1] = pY, pZ] + px, [dx, d y , ] (4.16) Solving the equation array gives A= pZ, and substituting for the term / gives: u + pza - px + pzT - p dx (4.17) d or r + (dpY- dp) = 0 dY u -dx. v + (pzdY)a - (pzdx) (4.18) which can be rewritten as the linear constraint [A1 , B 1 , k 1A1 , k 1B1 , C1 ]. Since all rays should satisfy this constraint, there must exist a and 3 so that aFll + 1r12= [Al,B1,klA 1,klB 1 ,C1 ] (4.19) Solving 4.19 gives the first characteristic equation of the GLC. 4.4.3 GLC Degrees of Freedom The canonical GLC representation defines any GLC using 6 free parameters. However, it does not mean that all GLCs have 6 degree of freedom (DOF). Consider that a pinhole camera can be uniquely determined by only three degrees of freedom (the x, y, and z coordinates of the center of projection). Thus, a pinhole camera must be subject to additional 3 constraints. This is not surprising, because pinhole GLCs are subject to the second characteristic equation (the edge-parallel condition). In general, the sum of the degrees of free (DOF) of a GLC and the number of 63 Table 4.2: Degree of Freedom of GLCs GLC Type P O TO PB X PN B EPI 4 DOF 3 2 5 6 3 5 6 1 pt 2 inf lines 1 line,1 pt 1 line 1 line 0 lines A plane 2 lines Geometry A=0 0 A=B=C=0 Constraints EP EP,A=0 A=O A=B=0 0 2 1 0 4 1 0 3 Equations 3 constraints it satisfies should be 6. In order to determine the DOF for a specific GLC, we only need to count the number of constraints. These constraints are from the first and the second characteristic equations, which constrain the coefficients A, B, C and the discriminant A, as well as the edge-parallel equation. For instance, a twisted orthographic camera is subject A = 0 and B = 0, thus it has 6 - 2 = 4 DOF. The complete characterization 4.5 of all GLC types is shown in Table 4.2. Pluicker Coordinates The major reason to choose a two-plane-parameterization to represent rays is that, under 2PP, rays form a vector space, whereas other ray/line parameterizations such as the Pluicker Coordinates, do not. Nevertheless, in this Section, we compare GLCs under 2PP to their equivalent structures under Plucker Coordinates. The Pluicker Coordinate P = [px, (PC) of a line I can be found by picking two points py, pZ] and Q = [qX,qY, qZ]on 1. The Pluicker Coordinate is the the six tuple 64 IPC = [10, 71/, 712, 13, 7r14 , 1'/5 ], where r1o = pqY - qpY 71 pqZ - (4.20) q pZ 7r12 = pYq -qyp 13 = pX - qX r71 = 4 pY - qY 7 15 = pZ _ qZ Plucker Coordinate has both advantages and disadvantages compared with 2PP. 2PP cannot represent rays that are parallel to the parametrization plane while PC can represent all possible rays. Furthermore, lines parameterized under PC are oriented, and thus preserve the direction of the ray, (P - Q or Q - P) while 2PP does not. PC also has other nice linear properties to determine the spatial relationship between the rays, as I will show later in this section. The major disadvantage of PC is that it does not form a vector space. More precisely, all rays under Plucker Coordinates lie on a bilinear manifold (notice the first three coordinates 7rlo,7r11 , 12 are bilinear functions of 713,714 , and 7rl5 ), and, therefore, the linear combination of the PC of two rays is not guaranteed to be a valid line/ray parametrization. Since GLCs are defined in terms of the affine combinations of rays, Plucker Coordinate is, thus, not suitable for representing rays in GLCs. The side operator of the PC is defined to determine the spatial relationship of two rays, i.e., if they intersect, are parallel, or oblique, as follows. Given two rays parameterized under Plucker Coordinate a[ao,al, a 2 , a3 , a 4 , a5] and r[ro, r1 , r2 , r 3 , r4 , r], we define side(a,r) = aor5 + ar 4 + a2 r3 + a3 r2 + a4rl + asro 5 (4.21) It has been shown in [33] that side(a, r)= 0 if and only if the two rays are coplanar (i.e., either intersect or are parallel). I use this property to simplify the proof of 65 Theorem 3.3-1. To do so, we first transform the 2PP parametrization of the ray to its Plucker Coordinate. Given a ray s, t, u, v] under 2PP, we choose two points P[s, t, 1] and Q[u, v, 0] and compute its Pluicker Coordinate using equation 4.20 as. r2pp[s,t, u, v] = rpc[tu - sv, u, v, u - s, v - t,-1] (4.22) Notice the middle four terms correspond to the relative-two-plane-parametrization of the ray, and the first term is bilinear in the other four terms. A GLC is defined as all possible affine combinations of the three generator rays as r[s, t, u, v] = a[si,t, Ul,vl] + /[S2, t 2 , u2 ,v 2] + (1 - a )[s3, t3 , U3,v 3] - All these rays can also be parameterized under Plucker Coordinate using equation 4.20 as rpc = [(atl + ft 2 + (1 - a- )t3)(aUl + /3u 2 + (1 - a - (as1+ S2+ (1 -a- f)s3 )(av + V2+ (1-a-3) aUl + fU 2 + (1 - a -/)U 3, )U3)- (4.23) 3), aV1 + 3V2 + (1 - a - 3)V 3, (aUl+ U2 + (1 - a - )U3 - (as1 + s 2 + (1 - a - /)S3), av1 +-/V 2 +(1-a-)v 3 - (at +ft 2 (1-a- f)t3 ),-1] Now we can easily prove Theorem 3.3-1 using PC and the side operator. Theorem 4.5-1. Given a non-EPI, non-pinhole GLC, if all camera rays pass through some line 1, not at infinity, in 3D space, then 1 must be parallel to u. Proof: Suppose all rays in a GLC pass through some line a that is not parallel to the parametrization plane. First we represent all rays in the GLC as rpc = [ro,rl,r 2 , r 3, r4 , rs] as shown in equation 4.22. Notice, r has both quadratic and bilinear terms in a and . It is easy 66 to compute the coefficientsA, B, and C of these terms as: 2 P1 = (tl - t3)(Ul oa 12 P2 = (t2 - t)(u 2 a/3: P3 = P l+P2-((t 3) - U3 ) - (82 - 3)(v2-t (4.24) - (1 -S3)(Vl - V3) 2 )(Ul-U 2 )-(S1 3) -S 2 )(v1 -v2) The remaining terms of the PC rl to r 5 are linear in a and 1. Now that consider if all rays in a GLC pass through a line a that is not parallel to the parametrization plane. If we also represent a under Plucker Coordinate [ao, a1 , a 2 ,a 3 , a4 , a5], we have a5 aplucker = 0. Since all rays in the GLC pass through a, we must have side(a,r) = 0 for all a and /. Computing side(a, r) using equation 4.21, it is easy to see a 5ro is the only term that consists of a 2 , a, and 32. And, since side(a, r) = 0 for all a and and a 5 $ 0, we must have the coefficient of the quadratic terms be zero, i.e, P1 = P2 = P3 = 0. This yields to the edge-parallel condition, which can only be satisfied by pinhole, orthographic, and EPI cameras. Furthermore, if a is parameterized using the intersections with 2PP as [uo, vo, 0] and [s, t, 1], its Plucker Coordinate can be computed as equation 4.22. We can then rewrite side(a, r) = 0 as (-1)(touo- soVo)+ (o)(v - t) + (v)(u- s) (4.25) +(uo - so)(v)+ (vo- to)(u)+ (-1)(tu- sv) = (s - so)(v- vo)- (t - to)(u- o) = Gu et al [12]derived this same constraint through other means. If a is parallel to 2PP, we pick two points on the slit as P[xi, Yi, zo] and Q[x 2 , Y2, o] so that it has PC as apc = [xly2 -x 2y 1, (X1 -X 2 )ZO,(Y1 -y 2 )Zo,( 67 -x 2 ), (Y1 -Y2),O] (4.26) Thus, side(a, r) to a linear constraint: (YI - y2)u +(x1 - X2)v +(Y1- y2)zo(u- ) +(X1 - X2)Z(V-t) + (XlY 2 - X2 Y1) = (Y1-y 2 )(1 + O) u + (x1 -x 2 )(1 + O) v -(Y1-Y2 )ZO' - (i -X 2)ZO t + (lY 2 -X2Y) =0 We can rewrite this linear constraint as As+Bt+KAu+KBv+C as [A, B, K (4.27) = 0, or abbreviated A, K. B, C], resulting in a consistent relationship as derived in Section 4.3. 4.6 Discussion To conclude, I have derived the relative two-plane-parametrization (R-2PP) in order to simplify the GLC characteristic equations. The first characteristic equation reveals the change of the area formed by the three generator rays on a sweeping plane. I have also explored some of the algebraic properties of the GLC model. Specifically, I have shown that GLCs are 2D linear subspaces in the 4D ray space and can be alternatively represented by two linear constraints. The two linear constraints satisfy specific algebraic properties for different types of GLCs. Finally, I have compared GLCs defined under two-plane-parametrization and under Pluicker Coordinates. 68 Chapter 5 GLC Projection and Collineation In this chapter, I present theories of projection and collineation in GLC images. The GLC imaging process can be broken down into two separate stages: the mapping of 3D geometry to rays and and the sampling of those rays over an image plane. In this chapter, I derive a closed-form solution to projecting 3D points in a scene to rays in a GLC. The GLC image is created by sampling these rays over image planes. This chapter develops a notion of GLC collineation analogous to pinhole cameras. GLC collirneation describes the transformation between the images of a single GLC due to changes in sampling and image plane selection. Given a GLC camera with two different image planes, the collineation maps the pixels on one image to the corresponding pixels of the second image. I show that general GLC collineations can be characterized by a quartic (4th order) rational function. GLC projection and collineation provides a basis for developing new computer vision algorithms suitable for analysing a wider range of imaging systems than current methods, based on simple pinhole projection models, permit. 5.1 GLC Projection GLC projection is the mapping of points in 3D to their corresponding ray in a specific GLC. First, we consider projecting 3D point onto a GLC. To simplify the analysis, I will use the canonical GLC representations (equation 4.6) with the three generator 69 rays [al 1 l,, 0, 0, [2, T2, 1, 01],and [a3, r3, 0, 1]. This setup describe almost all GLCs (except for a subspace of GLCs whose slits lie on the uv plane). Every ray [, r, u, v] in the GLC is subject to the linear constraint equations 4.8: J= (1- u - v)a 1 +ua 2 + a3 u- V)T1 + UT2 + VT3 T = (1- The GLC ray that passes through a point P(x, y, z) in 3D satisfy the linear constraints equation 4.10: U+z-a = x v+z- = y Equation 4.8 and 4.10 can be combined into a matrix form as U erl - Or2 rl--W2 a - 3 1 0 T1 -T 3 0 1 al - 2 - V T1 - T2 a 1 0 z 0 X T 0 1 0 z Y 3 T1 - T2 -T (5.1) The ray passing through P is, thus, the solutions of the four equations given in 5.1 as: -(Z2(alT3 - Z(l(y -1)- a3y - (T1- T3)) - a3T) - X) z2(si(t2 V 2 (5.2) t3) + s2(ta- tl) +S3(tl- t2))- Z(S1 - 2+ tl - t3)+ 1 . (z2 (alT2 - a2 T) - z(aly - 2y + T(1 - X) + T2X) + y) Z2(Sl(t2-t3) + s2(t3 - tl) + S3(tl - t2)) - Z(S1 - 2 + tl - t3 ) + 1 Notice the denominators in u and v corresponds to the characteristic equation of the canonical GLC as shown in Chapter 4. I call this equation the GLC Projection Equation. 70 F r3 (b) (a) Figure 5-1: (a) Projecting a point P to a ray in the GLC. (b) The projection of P can be computed using the same affine coordinate on the sweeping plane II,. 5.1.1 Plane Sweeping The GLC Projection Equation has an intuitive geometric interpretation. Consider a plane Hz parallel to the uv plane and passing through P. The three generators will intersect IIz at T 1 ,T 2 ,T 3 , where = (0,0,0) + z. (01, T-i, 1) = (az, rz, z) = (1,0,0) + z (O2, T2, 1) = (o2 z + 1, -72, z) T3 = (0,1,0) + z - (a3, T3, 1) = (a3 z, r3 Z + 1, z) The affine combination [a, /] of the three generator rays that passes through P, is: P = (1 - a - ) -T + a T2 +/3 T 3 (5.3) [a, /] can be computed using the ratio of the signed areas formed by triangle AtPlPT3 7,AlblT 2 P over AT 1T 2T 3 , as is shown in Figure 5-1. Notice the area formed by AT1T2T3 corresponds to the first characteristic equation of the GLC. Thus, the affine coefficients (, rl t 3) can be computed as: - -.- - At 1 PT3 ZO 1 ZT1 1 X y 1 1 + ZT3 za3 --. AZ2 + Bz + C ZO1 1 +za AT1 T2P A\T T 1 x 2 (r, A\ "' ZT 1 1 ZT2 1 y 1 Az2 + Bz + C ~ Equation system 5.3 and Figure 5-1 gives a geometric interpretation to equation 5.1. 5.1.2 Singularities Notice Equation 5.4 may lead to no solution or multiple solutions when the denominator Az 2 + Bz + C = 0 (i.e., the first characteristic equation is zero). This happens when P lies at the depth of a slit. Thus, using Table 3.1, we can conclude that these singularities can only happen in cross-slits, pushbroom, pencil, and pinhole cameras. When the points lie precisely on the slits, duplicated images will occur, because multiple GLC rays pass through these points. The ray passing through the point is determined by the solution to the 4x4 system of equations given in 5.1. When the point lies on the slit, the determinant of this matrix is zero, and, therefore, the four equations become linearly dependent. For pinhole cameras, when the point coincides with the center of projection, the 4 linear equations will degenerate to 2 linear equations as 4.10 and the projection of the point will cover the whole image. For pushbroom, cross-slits, and pencils, the 4 linear constraints will degenerate to three independent equations, and the projection of each point on the singularity covers a 1I) subspace of rays, or in its image it will project to a line. A similar case happens to EPI cameras. Furthermore, not all 3D points project onto a given GLC. There are two possible un-projectable situations: 1) there is no ray in the camera that can pass through 72 ·t<2 (a) (b) (c) "U) , Figure 5-2: (a) and (c) are cross-slits images synthesized from light fields. (b) and (d) are pushbroom images synthesized from light fields. The distortions of the curved isolines on the objects illustrate various multi-perspective effects of GLC cameras. The pushbroom camera also exhibits apparent stretching/duplications of objects faraway from the image plane, e.g., the lamp on the background. Table 5-1: Singularities in GLC Projections GLCType/ P0 Singularity Finite Pt. with Duplicated ot-ges (OP None Pts. onthe slit the p (OP and parallel tothe itage _ X _ PN T B EPI Pts.on the slits Pts. onthe slit None N.one the EPI plane _ _ _ __ _ _ _ _ P. ___________ P Pts. Unprojectable Ipassing tough ble FinitePt PB Pt. None ita pllel nd .e. . Projectahle Ifiit Pt.Intinltc ll PProjectabl itttageplane. Plo.withthe directtt as oroaphic _ _ _ direction the p passi pasing th li atd paallel to the Pts.onthe plane tg itheh through neof split Pts. on theplane throutagh the passing ite None the on image ~~~~~~~~~~~~both plane. slits.. ad plelt slit andparllel to Pts. that do None notlic tha donle oP. the EPI plane. Pts.withthe Pts. wtth the directiott ,.th parallel to the dieeetioo as the twisted All Projectabhile All Projectable ~~orthographicpln ~~~~~pushbroom plane otgrphic. Pt9. All Projectable Pts.onthe EPI PneI GLC Projections for Pinhole (P), Orthographic (), Pushbroom (PB), Cross-slits (X), Pencil (PN), Twisted Orthographic (T), Bilinear (B), and EPI. the point, or 2) the ray that passes through the point is parallel to 2PP, and hence cannot be represented. Points that cannot be projected can only happen when the denominator of equation 5.4 is zero and the numerator is non-zero. For cross-slit cameras, these points lie on the two planes HIIz=zl and IZ=Z 2 that contain the slits but do not; lie on these slits. This is representative of the first case. For pencil and pushbroom cameras, these singularity points lie on plane II,=, that contains the slit but do not lie on the slit, and it followsthe second case. Pinhole cameras are a special case of pencil cameras. In theory, it can image all points in 3D space. However, for points that lie on the plane parallel to the parametrization plane and passing through the COP, the corresponding rays are parallel to 2PP and hence cannot be imaged by a pinhole GLC, as shown in Table 5-1. 5.1.3 Projections of Lines Now we consider the projections of lines onto various GLCs. If uv plane, we can parameterize is parallel to the as a point [x0 ,y0o, 0] on the line and the direction [dx, dy , 0] of the line. It has been shown [12] that all rays passing through 1 satisfy [u,v, 0] + Al[a, , 1] = [x0 , yo,z 0 ] + A 2 [dx, dy, 0] 74 (5.5) which is equivalent to the linear constraint (u+ zoo -xo)dY - (v + zor -yo)dx = 0 (5.6) The GLC rays passing through is the intersection of three linear constraints: equation system 4.8 and 5.6. Thus, the rays collected by any GLC passing through 1 are, in general, a D linear manifold. If we assume the uv plane is the default image plane, then (u, v) gives the pixel coordinates of the projection. This implies that the image of a line 1 parallel to the uv plane also a line. If is not parallel to the uv plane, then 1 will intersect uv plane at (u0 , vo,0) and has direction (o, to, 1). All rays passing through in this case satisfy the bilinear constraint [12]: (u - uo)( - o)- (v - vo)( - o) = 0 (5.7) The projection of 1 hence can be computed using equation array 4.8 and equation 5.7 as follows: (u - uo)(( -u-v)ri +ur2 + vr3 - o) -(v - Vo)((1- u - v)o'1 +uo'2 +v which corresponds to a 3 - a0 (5.8) ) =0 D quadratic manifold of rays. Similarly, if we take the uv plane as the image plane, the image of is a quadratic curve on the image plane as shown in Figure 5-2(a) and 5-2(b). 5.1.4 Projections of Points at Infinity We can use the line projection to determine the GLC projections of points lying in the plane at infinity. An infinite point can be written as: P(x, y, z) = (o, vo, 0) + z(a0 , TO, 1), z -* oc 75 (5.9) Substituting P in equation 5.4, it is easy to see that the numerator and the denominator of u and v are both quadratic in z. Howevereither or both terms may degenerate to linear or constant. For pinhole, pencil, bilinear, and cross-slits, the first GLC characteristic equation (the denomrninatorin the projection equation) is always general quadratic in z, as is shown in Table 3-1. And since the numerator is at most a quadratic z- in z, when oc, both u and v will have finite values, i.e., points infinitely far away from the image plane all have a projection in the camera. Substituting [x, y, z] in equation 5.9 into 5.1, we get Auz 2 + Buz + Cu tl)+ 3(tl-t 2))- Z(sl- 2+tl -t 3) + 1(5'1 10)) U... z2(S1(t 2 - t3) + s2(t3 - Avz2 + Bvz + Cv V Z2(S(t2 - t3) +s2(t3 - tl)+s3(tl- t2))- Z(S1 - 82+tl- t3) +1 2'--: where (1 Au= T1 1 o 1 ,A= ao a3 a 3 1 a2 ao0 Tr1 1 2 1 ro0 1 (5.11) Thus, the coefficients of z2 in both the numerator and denominator of the projection equation are functions of ao, To, and the intrinsic parameters of the GLC, not uo or vo. This implies the final projection is only dependent on the direction of the infinite points. In the pinhole case, these points correspond to the vanishing points associated with directions and such vanishing points also exist for all pencil, bilinear, and cross-slits cameras. For pushbroom cameras, the directions of three generator rays are parallel to some plane Hpushboomand its characteristic equation is linear in z. The denominator in the Projection Equation 5.4 is, thus, a linear function in z. However, the numerator can 76 be quadratic in z in format as shown in equation 5.11. Therefore, only when l Au= eo 0a3 T1 1 l o 1 =0, T3 A = 1 T1 1 0r2 T2 1 =0 uo 0 To (5.12) 1 can the point be projected into the camera. However, since the three generator rays are parallel to some plane, we must also have a 1 T e2 2 1 U3 3 1 = 0 From equation 5.11 and 5.12, [ 0 , To,1] must be a direction parallel to (5.13) 11 ushom Thus the projection of the infinite points are constrained to one dimensional subspace and causes infinite stretching at the other, as shown in the pushbroom panorama Figure 2-3(d). Cross-slit GLCs, however, are able to project all points infinitely far away and, therefore, are a better choice for creating panoramas. Figure 5-2(a) and 5-2(c) compares pushbroom and cross-slit panoramas. Objects far away are stretched in pushbroonL cameras, but not in cross-slit cameras. Similarly, for orthographic and twisted orthographic cameras, whose first characteristic equations are constant, an infinite point has a projection only if [o, o, 0 1] is the direction of the ray of the GLC at point [u0,v0,0]. For instance, for orthographic cameras, only infinite points along the view direction can be seen in the projection. The complete characterization of singularity and infinite projection of GLCs is shown in Table 5-1. 5.2 GLC Collineation In the previous section, I study the projections of points and lines in a GLC by taking the uv plane as its default image plane. In this section, I derive how a given GLC 77 / /r7 / / ~ - (a) (C) (D) (d) Figure 5-3: (a) Imaging the GLC on a plane parallel to the uv plane. (b) Imaging the GLC on a different plane. (c) A GLC image of a toy dog synthesized from a video using image plane (a). (d) A different image of the same GLC camera using image plane (b). image transforms as it is sampled over different image planes. This transformation is analogous to planar collineation to pinhole cameras. Assume the new image plane lnew is specified by an origin P and two spanning directions dl, d2. For every ray r[aU , u, v] parameterized under the default 2PP, we intersect the ray r with Inew to obtain its new pixel coordinate [il, jl]: [u,v, 0] + A[, r, 1] = P + dlil + d2jl (5.14) To simplify our computation, we translate origin P on the plane to simply our computation. If we assume the new image plane is not parallel to the z axis, we can choose = [0, 0 ,pZ]. Otherwise, we can choose the image plane's intersections with either the x or y axis, and similar results hold. The new image coordinate of the ray [i1 , jl] can be computed as by solving equation 5.11 as: pZ(d2o- d2r)+ d'u - d2v+ d(v -Tu) 2 il~~~~~~ (5.15) (5.15) -y pz(d~-d) +dv-dlu- d(v - -ru) where y= dx dl d d2 dz dz a r (5.16) 1 Since all rays lie in a GLC satisfy equation 4.8, thus, a and r are linear functions in u and v.. Furthermore, -yis linear in a and , and, therefore, can also be written as y = a3 u + b3 v + C3 . Thus, il and jl can be computed as: il alu2 +bluv+clv2 +d u+e v+fl fl l (5.17) -(5.17) a2 u2 +b2 uv+c2 v2 +d2 u+e2 v+ f 2 ly -y is zero when the ray is parallel to the new parametrization plane, which we will not consider. The numerator is, in general, a quadratic function in u2 , v 2 , uv, and 79 therefore, the collineation from the default image plane (uv plane) to a new image plane is a quadratic rational. 5.2.1 General Collineation between GLCs Finally, we can compute the collineation between any two planar parameterizations of the same GLC. Suppose we want to compute the collineation from image plane ri1 = {, , d2 } with pixel coordinate [i1 ,jl] to image plane 112 = {qd 3 , d4} with pixel coordinate [i2 , j2]. First, we compute the transformation from 11 to the default image plane. Second we use the transformation from the default image plane to 112. The first transformation is in fact simply projecting each point on 11 onto the _~~~~ default plane, which can be described by the Projection equation 5.4. Consider each pixel m[ul, vi] corresponds to point M px+ildl+jd1 2 M[x,y,z] =P+d+1 i1 +d 2j 1 p +ild¥ +jld2 (5.18) P1+ ildl + jld2 The projection of M onto the uv plane is quadratic rational in [x, y, z] and [x, y, z] is linear in [i1 , jl] as shown in equation 5.13, therefore, the composed transformation can written as u = fi(il,jl) v 0(ii,ji) f2 (il,jl) (5.19) 0(ii,ji) where f1 (i1 , jl) = A1i 2 + B1iljl + Clj 2 + Dil + E1 jl + F1 (5.20) f 2 (il,jl) = A2 i2 + B2iljl + C2j 2 + D2il + E2j + F2 O(ii,ji) = A3i2 + B3 i 1jl + C3 j 2 + D3i1 + E3 j1 + F3 Thus the transformation from [u, v] is quadratic rational in i and jil. Next, we compute the Collineation from [u, v] to II2[i2 , j2] using equation 5.6 and result is a composition of two quadratic rational functions, which is a quartic rational 80 function that has the form: 2 Hl(il,jl) I(i, j ) i2 where Hi(il,jl), (5.21)j) H2(il,j) (5.21) I(i, j) ' H2 (ii,jl) and I(il,j 1 ) are all quartic (4th order) polynomials in il and j. In Figure 5-3, we show two images (in red rectangle) of the same GLC camera using two image planes. The GLC used is a cross-slit. This indicates, although the collineation is quartic rational, the behavior of changing the image plane is similar to homography in a pinhole image. 5.2.2 Collineation Degeneracies The planar collineation of a GLC is in general a quartic rational function. However, it may degenerate into a quadratic or even a linear rational function for specific types of GLCs. In this section, the form of special collineations for cross-slit, pushbroom, and pinhole cameras are considered. First, consider equation 5.15 which maps rays onto an image plane IIl as pixel [i1,jl], as a function of u, v, a, and . Given a second image plane II2, we can represent u, v, a, and r in terms of its pixel coordinate [i2,j2], and directly compute the collineation from II2 to ll. If 112is parallel to the default 2PP, then u, v, a, and T are linear functions of i 2 and j2, as shown in Chapter 4. Substituting them into equation 5.15 gives a quadratic rational (quadratic in the numerator and linear in the denominator) collineation. If 112 is not parallel to the default 2PP, then it is non-trivial to represent a, r, u, and v in terms of i2 and j2, because ao, , u, and v are no longer linear functions of i 2 and j2. However, for some special types of GLCs, such as pinhole, pushbroom, and cross-slits, it is still possible to represent a, For cross-slit cameras, if 112 T, u, and v in simpler forms of i2 and j2. is not parallel to the default 2PP, then the two slits will intersect the new 2PP can be represented as two lines 11[al, rl, ul, vi] and 12[a 2 , T2 , u2.:, v2]. If we choose 2 as the image plane for parameterizing 81 each ray [a, T, u, v], we have u = i 2 and v = j2. Furthermore, since all rays passing through 11 and 12 are subject to two bilinear constraints as is shown in equation 3.3. Thus, (i2 - ul)(r - T) - (j2- v)(a - a 1) = 0 (i2 - 2)( we can solve a and T 2) (5.22) - (j2 -v2)(a - 2)= 0 in term of i 2 and j2 as: A 2i 2 + B 2 i2 j 2 + C2j2 + D2 i 2 + E 2j 2 + F2 a (523) A 1 i2 + Bj2 + C1 A 3 i2 + B 3 i 2j 2 + C3j2 + D3i 2 + E 3 j 2 + F3 A l i2 + Blj 2 + C1 T - where Ai, Bi, Ci, Di, Ei, Fi, i = 1, 2, 3 are all constant. Expanding equation 3.3 gives i2r - Finally, if we substitute 2a = i2T -j 2 A 4 i2 + B4 j 2 + C4 cr + D4 T + E4 a, (5.24) a, and r into the collineation equation 5.15, the quartic rational collineation degenerates to quadratic rational (quadratic in numerator and denominator). An alternative proof using a quadratic tensor is shown in [47]. For pushbroom cameras, if 2 is not parallel to the default 2PP, then the slit will intersect the new 2PP as the line [a, T1 , u 1 , v2], and the new ray parametrization [a,r, i 2 , j2] is subject to one bilinear constraint, as is the case for cross-slit GLCs: (i2 -u)(r - T)-(j 2 -vl)( -) =0 (5.25) In addition, since all rays of a pushbroom GLC are parallel to some plane, all rays mustalso satisfy a second constraint [a, r, 1]. [no,n1, n2] = noa + n1t + n2 82 (5.26) Solving a and r for equation 5.22 and 5.23 gives (5.27) A 2 i2 + B 2j 2 + C2 Ali 2 + Blj 2 + C1 A3i2 + B3j 2 + 3 Ali 2 + B1 j2 + C1 in the collineation equation, and Similarly, we can substitute i2T - j 2 a, a and the quartic rational collineation degenerates to a quadratic rational (quadratic in numerator and linear in denominator). For pinhole cameras, rays in the GLC satisfy We can solve a and T- as a = -Plzi2 + 5 and i2 + pZa = pX and j2 + PZT = pY. = pl 2 + . Substituting a and T into the collineation equation, the quartic rational collineation degenerates to a linear rational (a homography). Finally., for bilinear, pencil, twisted-orthographic GLCs, when the first image plane is not parallel to the default 2PP, it is non-trivial to solve a and T in terms of i2 and j2 in general. And the quartic rational collineation will not reduce into simpler form. In summary, I have presented theories of projection and collineation for GLC images. A closed-form solution for projecting 3D points to rays in a GLC was derived. In addition, I showed that colllneations between the images of the same GLC but on different imaging planes are up to quartic (4th order) rational. Using GLC projection and collineation theory, it is easy to explain the multiperspective distortions such as curving of lines, apparent stretching and shrinking, and duplicated projections of a single point. 83 84 Chapter 6 GLC Differential Invariance Theorem In this chapter, I will expand the use of GLCs to model arbitrary multiperspective cameras. first introduce the model of an arbitrary smooth (non-planar) ray manifold that maps rays collected by a multiperspective camera onto a two-dimensional subspace in the ray space. I then show how to model this ray manifold by analyzing its local tangent planes. Each local tangent plane corresponds to a GLC. The GLC's type can be used to characterize the group behavior of around each ray. The set of rays in a GLC, however, are not invariant to the choice of the parametrization plane. Therefore, in this chapter I will prove a GLC differential invariance theorem that states that, even though the ray manifold changes when the parametrization plane changes, the GLC type and geometric features remain invariant. 6.1 Limitations of 2PP A multiperspective camera in general captures a 2D subspace of rays that lie on a 2D (non-planar) manifold in the ray space under two-plane-parametrization (2PP). When we choose a different set of the parametrization planes, the same set of rays will lie on a different 2D ray manifold. As a special case, all GLCs that correspond to 2D linear subspaces of rays under the default 2PP may map to non-linear manifolds 85 under a different choice of the parametrization plane. For instance, consider a cross-slit GLC camera under a given 2PP. All rays in this cross-slit GLC lie on a 2D hyperplane as we have shown in Chapter 3. Now if we choose a different set of parametrization planes, not parallel to the original 2PP, I will show the same set of rays lie on a non-linear manifold under the new parametrization. Since the new parametrization planes are not parallel to the original 2PP, each of the two slits must intersect with the new planes at two points [si, t, 1], [ui, v, 0], i = 1, 2. Recall that all rays in cross-slits pass through these two lines and are subject to two bilinear constraints under the new parametrization (s-s~)(v-vl) - (t-tl)(u- -2)(v - V2)- (t - t2)- as shown in Chapter 4: Ul) = O 0 (6.1) = Since the intersection of two bilinear subspaces is, in general, a non-linear manifold, the same set of rays of one GLC lie on a non-linear manifold under the new parametrization whereas they lie on a 2D linear subspace under the original parametrization. In general, with the exceptions of pinhole, orthographic, and EPI GLCs, all other GLCs map to non-linear manifolds when the new parametrization plane is not parallel to the defining 2PP. This is because the bilinear surface formed by the two generator rays shown in Chapter 3 changes when the parametrization plane changes. In addition, the local tangent plane on an arbitrary ray manifold at the same ray will also be different under this new parametrization. An intrinsic ray analysis should reveal the behavior of rays independent of parametrization. I show that differentially, in a infinitesimal region about each ray, GLC analysis satisfies this invariant property. For instance, although a cross-slit maps to a non-linear manifold when the new 2PP is not parallel to the default 2PP, the local tangent plane around any ray on this new non-linear ray manifold, is still a cross-slit camera. 86 6.2 Local Tangent GLC Given 2D continuous ray manifold under [a, r, u, v] parametrization, it can be rewritten in term of two free variables, e.g., x and y as (x, ) = [(x, y), (x, ),u(x, ), (x,y)] (6.2) where u(x, y), T(x, y), u(x, y), and v(x, y) are continuous and differentiable in x and y. We can compute the tangent plane of this manifold by computing two spanning vectors dc and d2 at each point on the manifold by taking partial derivatives with respect to x and y as = [Trx ,uX,vx],d2 = [y, y, uy, vy] (6.3) This is the same as finding the tangent plane of a surface in 3D space. There are two significant properties of the ray manifold's tangent planes. First, they characterize group behavior of the neighboring rays around each point on the manifold. Second, since a tangent plane is a 2D linear manifold, it must be one of the GLCs. The group characteristic behavior of the rays on the tangent plane is determined by the intrinsics of the GLC camera type as determined by its characteristic equations. To compute the GLC characteristic equations of a tangent plane, we can choose three points on the tangent plane, which are E(x, y), E(x, y) + d1 , and E(x, y) + d2 , yielding the following first characteristic equation: u+ A v + A 1 (u + u)+ (a+a x) (v+vx)+A( + x) 1 =0 (u+uy)+A(a+ay) (v+vy)+A(T+Ty) 1 87 (6.4) As usual, it corresponds to a quadratic equation AA2 + BA + C = 0 where A = acy-ayr B a-vy- = v - rTUy + Tryu (6.5) C = uzvy - uyv The discriminant, A, can be computed as A = (r - -r + 2 vy- %Vx) 4(axry - ay7-)(Uvy - Uyvx) (6.6) The second characteristic equation (edge-parallel condition) also simplifies to: 6.3 UxVx -TxUx O, (o - oy)(vx - O'yVy TyUy = O (6.7) vY) - (T - Y)(uX - uY) = 0 Differential Invariant Theorem Our goal in this section is to show that the first and the second characteristic equations of the local tangent GLC correspond to the same GLC type despite changes in the parametrization planes. Recall that the GLC type is determined by, A, B, and A in the first characteristic equation, and the edge-parallel condition. In order to prove the the differential invariance of GLC type, I show that the signs and zeros of the characteristic equation's A, B, and A terms, as well as the edge-parallel condition, are invariant to the parametrization. Given a set of rays that map to a 2D manifold under a specific parametrization, I first compute the transformation from the old manifold to the new one. Assume the new parametrization planes are specified by two points P5and on each of the plane, and two spanning directions d, d2 of the plane. For every ray [a, r, u, v] on the old 88 manifold, the intersections of the ray with the two new planes s', t'] and [u', v'] are given by: [u,v,]+Aj[a, , (6.8) 1] = P+d$'+d2t' + d' + d2v [u,v, 0]+A 2 [a,r, 1] = Theorem (3.3-2) shows that the characteristic equations of a GLC does not change when the origins P and translate in the plane. Assuming the two planes are not parallel to the z axis, we can choose P = [0, 0, pZ] and = [0,0, qZ] to simplify the analysis. ()therwise, we can choose their intersections with either the x or y axis and / similar results hold. The new [a', T', u', v'] can be computed as: / = s-U = (pz q)(d2ya - d2r) t'-v -(qZ pZ)(~la dir) 9-_ (-dxT)+ + ,s= q(a-d~ v - d T) +d - vy dudz(av- u) qZ(dlo -y where d Y= d 2d d2 a r dl (6.9) d 1 The new [', T', u', v'] parametrization is bilinear rational function of a, , The transformation , v. also has a singularity where y is zero. This happens when a ray in the original parametrization is parallel to the new parametrization plane. The spanning vectors of a tangent plane under the new parametrization can be computed by taking the partial derivations of [', T', u', v'] as [a', T, u', v'] and [a, T, u, t.y] using the chain rule. The new first and second characteristic equations can be computed using Equation 6.5 to 6.7. As a result, the relationship between the default characteristic coefficients A, B, 89 C and A and the new ones A', B', C' and A' satisfy: (dTd A' - d d) (dd2- B C =_(C72 2 Y3 (pz - qZ)2 A (6.10) - qz)(B?+ 2A) -d)(pz 2 73 + B7¢+ A2) 73 (dd2 - d2dY1)2 (pz - qz)2A 74 The edge-parallel equations can be computed as: =z - I (o I U = L(oxv - rxux) I I crv,u(ayvy('- -rUy) - TU/ y,)(v =- v,) - TY,)(U' u,) I ,((a ,y)(Vx vY) - (T- - 'ry)(uX - uY)) where (dxd2 - ddl)(pz /P = dxZ = - qZ) 272 dl (6.11) (6.11) dz d2 d2y d2 u v qz Since the t-Iwoplanes intersect the z axis at two different points p and q, pZ _ qz 4 0. The two spanning vectors on the parametrization dld2 - plane have to be different, so d2d 5 O. Thus, A'= 0 if and only if A = 0. And when A = 0, B' = 0 if and only if B =- 0. Similarly, when A = B = 0, C' = 0 if and only if C = 0. Furthermore, the sign of A' is always the same as A, and the new edge-parallel condition is satisfied if and only if the original one is satisfied. Thus, the local GLC type under the new parametrization is the same as under the original parametrization. In summary, I have shown the Differential Invariance Theorem that, differentially, 90 the local GLC model for an ray manifold is invariant to the choice of parametrization plane. This overcomes a major limitation of 2PP. I will use the GLC Differential Invariance Theorem to analyze reflection manifolds in Chapter 8 and to discuss the relationship between the local cross-slit GLC to the caustics in Chapter 8. 91 92 Chapter 7 Multiperspective Rendering In this chapter, I demonstrate the application of the GLC model for the direct rendering of multiperspective images. Multiperspective imaging systems are treated as devices for capturing a smoothly varying set of rays, and I show that the corresponding multiperspective images can be characterized as continuous manifolds in ray space. I then use GLCs, which describe all 2D linear manifolds of rays, as primitives for constructing mrultiperspective images. I show GLCs when constrained by an appropriate set of rules, can be laid out to tile the image plane and, hence, generate arbitrary multiperspective renderings. Finally, I show how the GLC framework can easily render a broad class of multiperspective images, such as multiperspective panoramas, neocubist style renderings, and faux-animations from still-life scenes. 7.1 Background and Previous Approaches A perspective rendering represents the spatial relationships of objects in a scene as they would appear from a single viewpoint. In contrast, a multiperspective rendering combines what is seen from several spatially incongruent viewpoints into a single image and still preserves spatial coherence, depicting, within a single context, details of a scene that are simultaneously inaccessible from a single view, yet easily interpretable by a viewer. Multiperspective rendering techniques are frequently employed by artists to de93 pict more than can be seen from any specific point. Classic examples include the visual paradoxes of Escher, and the Cubism of Picasso and Braque. Multiperspective images have also been used as backdrops in cel animation to effectively depict camera motion in a single panorama. Computer generated multiperspective panoramas, as presented by Wood et al [39], combined elements of multiple pinhole cameras into a single image using a semi-automatic image registration process. They relied on optimization techniques, as well as optical flow and blending transitions between views. The concentric mosaics of [32] and [27] are another type of multiperspective image that is useful for exploring captured environments. Durand [9] suggests that specifying multiperspective projections can also be an interactive process and uses them as an example to distinguish between picture generation and user interaction. Examples of such approaches include the 3D-based interactive rendering systems by Agrawala et al [1]and Hanson and Wernert [14]and the image-based approaches such as pushbroom and cross-slit panoromas and the MCOP (multiple center of projection) images of Rademacher [28]. 7.1.1 GLC as Multiperspective Images GLCs can be used to generate common multiperspective images such as pushbroom and cross-slit images. They also include many lesser known multiperspective cameras, such as the pencil, twisted orthographic, EPI, and bilinear cameras. Many of the GLC images exhibit typical multiperspective features such as shearing, twisting, curving, and duplications, as discussed in Chapter 5. An important characteristic of GLCs is that any general point in 3D space has a unique mapping to a ray. This is because under s, t, u, v] parametrization, all rays passing through a 3D point also lie on a 2D hyperplane and two hyperplanes (one for the point and one for the GLC), in general intersect at a unique point in 4D. Therefore, there is only one ray of a GLC that passes through a given point in a scene. GLCs, thus, are particularly useful for imaging systems since each 3D point has a unique mapping to the image plane. Furthermore, one can also choose three desired rays from different pinhole images 94 Figure 7-1: Image fusion by composing a GLC image: a new pose of the cow (right) is synthesized by specifying the three generator rays (red) of a GLC from left three poses. and synthesize a single GLC that incorporates the selected features. Collineation can be used to adjust the resulting image, to locally reduce the apparent distortions. Figure 7-1 shows a bilinear camera image that fuses the horn, tail, and nose of a ceramic cow from three different perspectives. In this Chapter, I use GLC models as rendering primitives for generating arbitrary multiperspective rendering effects. 7.2 Multiperspective Ray Manifold I treat the problem of multiperspective rendering as one of specifying and sampling a smooth varying set of rays. The model of smoothness that I define models each multiperspective image as a continuous 2D manifold of rays embedded in a 4D space. The continuity of the manifold guarantees that the images produced are coherent. Specifically, they are CO or higher-order continuous within a space of rays. The proposed approach is enabled by the local GLC model approximation method described in Chapter 6. Specifically, since GLCs model all 2-dimensional linear subspaces in Figure 7-2: A neocubism styled multiperspective image synthesized using our framework. It combines pinhole images from different points of views with various multiperspective (pushbroom and cross-slit) images. 96 (a) (d)ý (b) i· (g) v 1ý (h) Figure 7-3: General Linear Camera Models. There are precisely eight GLCs, shown in (a) to (h). (a) In a pinhole camera, all rays pass through a single point. (b) In an orthographic camera, all rays are parallel. (c) In a pushbroom, all rays lie on a set of parallel planes and pass through a line. (d) In a cross slit camera, all rays pass through two non-coplanar lines. (e) In a pencil camera, all coplanar rays originate from a point on a line and lie on a specific plane through the line. (f) In a twisted orthographic camera, all rays lie on parallel twisted planes and no rays intersect. (g) In an bilinear camera, no two rays are coplanar and no two rays intersect. (h) In an EPI camera, all rays lie on a 2D plane. ray space, they are first-order approximation (piecewise planar) of the continuous ray manifold shown in Figure 7-3. Thus, I use GLCs as primitives for constructing arbitrary multiperspective images. These atomic cameras, when constrained by an appropriate set of rules, can be laid out on tile the image plane, thereby generating arbitrary multiperspective renderings. In fact, to a first order, the GLC-based framework can describe any multiperspective image that is consistent with a smoothly varying set of rays. 7.2.1 Piecewise Multiperspective Image Plane While the GLC model includes many multiperspective camera types, it is still not sufficient to describe a general multiperspective image. Multiperspective GLCs respect specific linear geometric structures. For instance, they describe specific sets of rays that all pass through either a specific point, line, pair of lines, or lie on a ruled bilinear surface. Multiperspective rendering, as practiced by the masters, follow a - Generator Figure 7-4: A tessellation of the image plane with triangles that correspond to different types of general linear cameras. This model is equivalent to a 2D piecewise planar manifold embedded in 4D. far less restrictive set of rules. Specifically, they only attempt to preserve spatial continuity in the image plane, which can be accomplished by range of smooth, yet nonlinear, variations of viewpoint. I exploit a different aspect of GLCs in order to assure spatial continuity without restricting viewpoints to fall on specific linear structures. This characteristic of GLCs relates to the fact that a smooth variation in sample rays on the image plane is necessary to provide spatial continuity in rendering. Under a 2PP rays map to points in 4D. Therefore, any smooth 2D manifold embedded in 4D will correspond to rays of a multiperspective image. Imposing a parametrization over the manifold enables sampling and the subsequent rendering. Under this interpretation, GLCs describe all planar, or flat, manifolds in 4D. While this is a limited set of potential manifolds, it is sufficient to construct a piecewise linear approximation of any given manifold. In other words, any piecewise planar tessellation of a "ray-space" manifold corresponds a specific collection of GLCs, much like a polygonal model of a curved surface. An equivalent interpretation of the 4D points in 2PP is an image plane with an attached ray, much like a generator of a GLC. Therefore, a general class of multiperspective images can be described by a triangulation of the image plane along with generator rays attached to each vertex as shown in Figure 7-4. As the tessellation of the image plane increases, this model can approximate arbitrarily smooth 2D manifolds, and hence render arbitrary multiperspective images. When the triangles and generators of a given tessellation are analyzed according to their type, as determined by the characteristic equation 3.8, one finds that there are specific rules (constraints) for tiling an image plane using GLCs. 7.2.2 Adjacency Table Since each triangle on the image plane corresponds to a general linear camera, adjacent triangles sharing a common edge represent two GLC cameras that share two rays. This imposes a constraint on possible pairs of adjacent GLCs. For instance, a pinhole camera cannot share two rays with a different pinhole camera (because rays in two different pinhole cameras pass through two different points). Similarly, a pinhole camera cannot be adjacent to a bilinear camera, because any two rays will intersect in pinhole while no two rays will intersect in a bilinear camera as shown in Figure 7-3. Table 7.1 shows all possible adjacency relationships between general linear cameras on a C ° continuous manifold. Triangulations of the image plane into GLCs must satisfy these adjacency constrains in order to assure CO continuous images. Notice, the adjacency table is different from the transition table described in Chapter 6. The transition table describes the possible adjacency of the GLC on a C 1 continuous manifold and hence the adjacency is analyzed for a differential surface patch. The adjacency table for piecewise planar manifolds, describes if the two adjacent GLCs can share an edge, i.e., a D manifold of rays. For instance, in the table, no two 99 Table 7.1: Adjacency Tables of GLCs PB X PN T B Possible Adjacency P Pinhole(P) N N Y Y Y N N Orthographic(O) N N Y N N N N Pushbroom(PB) XSlit(X) Pencil(PN) Twisted Orthographic(T) Bilinear(B) Y Y Y N N Y N N Y N Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y different pinhole cameras can be adjacent to each other. Thus, there does not exist a multiperspective image which looks locally like a pinhole camera everywhere. However, there do exist multiperspective images which look locally like pushbroom or cross-slit images everywhere. In fact, multiperspective panoramas for cel animations are good examples of this type of multiperspective image. 7.3 GLC Multiperspective Rendering 7.3.1 Image Layout While any triangulation of the image plane with generator rays at each vertex describes describes a multiperspective rendering, it is not a very intuitive specification. In practice, I employ a multiperspective image design method that is inspired by the automatic layout method described by Wood [39], but with user guidance. The technique works as follows. A predefined triangular mesh is placed over the image plane. The user then guides any typical GLC image over the mesh to establish rough spatial relationships. The images can overlap as desired. The mesh then acquires generator rays by extracting rays from the reference images. If more than one image overlaps a vertex various blends of the rays from the reference images can be used, so long as the blend preserves affine combinations. The end result is that corresponding rays are interpolated such that the transition is smooth. I provide an interface that allows users to select from a palette of general linear 100 Figure 7-5: Desirable image fragments from different GLCs are overlayed on the image plane. 101 camera images. These images can be rendered directly from either a 3D model or a light field. Both 3D linear and 4D light fields can be used. The user can inspect and crop interesting regions from GLC images and then lay them out on the grid as desired. This process is shown in Figure 7-5. It is very useful to use irregular crops, which I call image fragments, to focus on the desired scene elements. When an image fragment of a general linear camera is placed onto the image plane, I first assign pixel coordinates to the fragment. I then compute the affine coordinates of all vertexes of the GLC triangulates inside the fragment and use the same weight to compute their [a, T, u, v] ray coordinates. The GLC rendering system allows the user to perform any transformation on image fragments that maintains affine relationships including translation, scaling and rotation. Each time a transformation is performed, the pixel coordinate of the fragment is recomputed and the ray coordinates for all vertexes of GLCs inside fragment are updated. Any triangle on the image plane whose three vertex-ray coordinates have been associated with a generator will form a GLC that can be rendered by ray tracing or querying into a light field. The image plane, therefore, embodies a piecewise linear multiperspective manifold and maintains Co continuity. 7.3.2 Ray Blending When multiple GLC image fragments are placed onto the image plane, the (s, t, u, v) ray coordinates of points lying in overlapped regions need to be computed in order to maintain a smooth transition from one general linear camera to another. To do so, the ray coordinates of each point in the overlapping region are found. The shortest distance from the point to each of the image fragments, i.e., the distance to the closet point in the corresponding fragment that is not in the overlapped region, as is shown in Figure 7-6. This distance is used as a weighing function to blend ray coordinates. Finally, all triangles are rendered as general linear cameras. Notice a triangle whose three vertexes come from a common reference image can be directly copied from the image fragment, thus saving processing. The complete algorithm is 102 I II I I I I i I GI-C Figure 7-6: Ray coordinate of each vertex in the overlapped region is computed using Algorithm 1. shown in Algorithm 1. 7.4 Multiperspective Sampling With regard to sampling, the GLC model is different than the specification of traditional camera models. A GLC determines only a set of rays seen, it does not uniquely specify how these rays are sampled across the image plane. Image plane sampling requires a consistent parametrization between GLCs of the multiperspective manifold and can be specified by a collineation as described in Chapter 5. In this case, a consistent parametrization is achieved by specifying a sampling plane with uniform grid and using the collineation between the sampling plane to the default image plane, as is shown in Figure 7-7(c). To understand this sampling approach it is helpful to tease apart a traditional pinhole camera into its separate components. A pinhole camera is specified by a center-of-projection, the point where all rays originate, and a sampling plane which 103 nage rigin Image Origin C t.,i S 21. ZI) 2, -UV. 2) Uz / (S,, ;ul, J1) Image Plane (c) Figure 7-7: (a) Sampling of a pinhole camera (homography). (b) Sampling of a cross-slit camera. (c) Sampling of an arbitrary multiperspective image is achieved by assigning a sampling plane to each GLC. 104 Algorithm 1 Compute Ray Coordinates rblend for each image fragment img do for each grid point P(i, j) in img do calculate the rimg(s, t, u, v) coordinate of P(i, j) in img add rimg to P's ray list Listr add img to P's image list Listimg end for end for for each grid point P(i,j) do if P's image list Listimg is not empty then for each image fragment img in Listimg do compute the distance disimg from P(i,j) to the closest point in img that is not overlapped compute the weighing factor weightimng in term of disimg end for normalize weighing factor rblend = (0, 0, 0, 0) for each image fragment Img in Listimg do rblend+ = rimg* weightimg end for end if end for is determined by an image origin and two vectors spanning the plane as shown in Figures 7-7(a). Integer multiples of the spanning vectors determine the sampling grid. The center-of-projection is the GLC component of the model, whereas the sampling plane and the grid implied by the spanning vectors is the sampling component of this model. The process of orienting the image plane in space has the effect of projectively warping the image without changing the set of rays seen by the camera, and is commonly referred to as homography in perspective cameras. In Chapter 5, I have shown collineation is up to a quartic rational function. Therefore, the sampling of a GLC can be modified by similarly transforming the sampling plane. However, each grid point on the sampling plane can be seen only once. Fortunately, all GLCs maintain this property except for at singular points for the particular GLC (COP of pinhole, two slits of cross-slit, all points of an EPI, and etc). In Figure 7-7(b), I show an example of the sampling scheme for a cross-slit camera. I call changes in the image due to the Collineation transformations of the 105 image plane perspective distortions, whereas I call distortions introduced by smooth changes in the center of projection projective distortions. 7.4.1 Perspective and Projective Distortions Once a generator ray is established for each vertex of the multiperspective image, one can render each pixel on the image plane by either ray tracing or querying a light field. Since pixels are uniformly sampled on image plane, rays are sampled uniformly over all the GLCs of the multiperspective manifold. Such uniform sampling, done independent of the projection, leads to noticeable projective distortions. For example, consider two pinhole cameras (not adjacent to each other) from a common multiperspective image, with congruent triangle regions, but whose centers-of-projection are at different depths. Imposing a common image plane has the effect that the camera with larger depth will have a smaller view of the scene than the closer viewpoint looking though an image-plane triangle of the same area. The mixing of very different fields of views leads to significant projective distortions in the final image. Our goal in sampling is to introduce subtle perspective distortions in order to smooth the transitions of projective distortion. An example of an image with an abrupt projective distortion is shown in Figure 7-8(a) and its correction is shown in 7-8(b). 7.4.2 Resampling I provide a simple algorithm to correct projective distortions by smoothly varying the sampling rate across the general linear cameras. I assume rays on each GLC are specified using a sampling plane, as is shown in Figure 7-7 and the sampling plane itself is uniformly sampled. When the sampling plane is close to the actual scene geometry, sampling the GLC rays uniformly will reduce the projective distortion effects. Thus, when a 3D model of the scene is available, we can simply align the sampling plane close to the real geometry. When a light field is used, we can estimate the depth of the scene in terms of disparities, and align the sampling plane closer to the geometries using a method similar to the focal plane used in dynamically 106 (b) (a) Figure 7-8: (a) A multiperspective image before sampling correction; (b) A multiperspective image after sampling correction using sampling planes is applied. In the original stuffed toy, the head is oriented such that it looks at the tail. This multiperspective image appears like a plausible perspective image with a different pose of the toy. reparametrized lightfield rendering [17]. Figure 7-8 compares multiperspective images of a stuffed toy rendered from a 360 degree turntable sequence. When we compose a different perspective of the head and tail of the stuffed toy to the body, we observe foreshortening artifacts in Figure 7-8(a), due to projective distortion. By estimating the depth of the head and the detail, we can then correct this distortion by resampling, as is shown in Figure 7-8(b). 7.5 Multiperspective Rendering Results The GLC framework focuses on the direct rendering of multiperspective images from either image-based or traditional 3D models. 7.5.1 Multiperspective Panoramas Panoramas attempt to combine a series of views into a single image that is viewed in pieces. Sets of GLC triangles can be constrained to match specific pinhole views while bands of transition GLCs can be used to fuse the images into a single multiperspective panorama. A practical advantage of our approach over that of Woods et al is that 107 (a) (b) Figure 7-9: (a) Nusch Eluard by Pablo Picasso; (b) Our synthesized multiperspective image. there is no need to compute optical flow or blend the overlap regions between images. Images generated using the GLC framework can be as sharp as the source material. Alternatively the images can even be rendered directly. In Figure 7-10, I show a multiperspective panorama synthesized from virtual flight path over a city. I also show the layout of the camera types on the right, where blue represents pinholes, red represents cross-slit and green represents pushbrooms. Although many adjacent cross-slit cameras are labelled as the same camera types, because of the smooth blending algorithm, they are different cross-slit cameras. 7.5.2 Artistic Multiperspective Rendering perspectives from multiple viewpoints can be combined in ways other than panoramas. By making subtle changes in viewing direction across the imaging plane it is possible to depict more of scene than could be seen from a single point of view. Such images differ from panoramas in that they are intended to be viewed as a whole. Neo-cubism is an example. Many of the works of Picasso are examples of such multiperspective images. In 108 Figure 7-9 compares one of Picasso's paintings with our synthesized image. Starting from a simple layout, it achieves similar multiperspective effects. In Figure 7-11, I show the mnultiperspective view of the teapot by overlaying image pieces from significantly different perspectives. It is possible to show close to 360 degree view of the teapot, reminiscent of an MCOP image [28]. 7.5.3 Multiperspective Faux-Animation It is possible to use multiperspective rendering to create fake or faux-animations from still-life scenes. This is particularly useful for animating image based models. In Figure 7-12, I show three frames from a synthesized animation. The top row shows the synthesized multiperspective images with strong perspective distortions. The bottom row significantly reduces this distortion using GLC collineation and resampling as discussed in Chapter 7.4.2. By using perspective correction, one can achieve plausible motions. Zomet [47] used similar approach by using single cross-slit camera (one of our general linear camera) to achieve rotation effects. To summarize, I have presented a framework for the direct rendering of multiperspective images by treating multiperspective imaging systems as continuous manifolds in ray space and using them to approximate piecewise planar sections, where each facet is a GLC. Multiperspective renderings can be easily achieved by laying out different pieces of GLCs on an image plane. Using the GLC framework, it is easy to achieve a broad class of multiperspective images. The GLC analysis reveals multiperspective images are constrained by neighboring tangent planes and hence cannot be arbitrary combinations of GLCs. I have also shown multiperspective images have both perspective and projective distortions, and by applying a spatially varying sampling plane, projective distortions can be significantly reduced. 109 (a) (b) Figure 7-10: Multiperspective panoramas: (a) Our synthesized multiperspective panorama from a virtual flight path. (b) Blended camera types of the multiperspective image. Different GLCs are marked with different colors, red for cross-slit, blue for pinhole, and green for pushbroom. 110 (a) (b) Figure 7-11: (a) A perspective view. (b) A synthesized multiperspective image that fuses different perspective views of the handle and beak. (c) A omni-perspective image that shows a 360 degree view of the teapot. (d) _ (e) Figure 7-12: From left to right: extracted images from a faux-animation. The source images were acquired by rotating a ceramic figure on a turn table. Multiperspective renderings were used to turn the head and hind quarters of the figure in a fake imagebased animation. Top row: without resampling. Bottom right: with resampling. 111 112 Chapter 8 Reflection Analysis In this chapter, I present a novel method to analyze reflections on arbitrary surfaces using GLCs. I model reflections as a continuous ray manifold and use GLCs to characterize the rays. I then provide an analytical framework to locally model reflections as specific GLC cameras. While classical approaches use local pinhole approximations, the GLC framework uses multiperspective camera models to better characterize the complicated image distortions seen on irregular mirror surfaces as well as the conventional catadioptric mirrors. Next, I show the connection between multiperspective camera models and caustic surfaces of reflections and demonstrate how they reveal important surface rulings of the caustics. Finally, I show how to use our analysis to assist mirror design and characterize distortions seen in catadioptric imaging systems. 8.1 Background and Previous Work Reflections are important visual phenomena, as well as a potential class of images suitable for analysis by computer vision. Although the basic physics and the geometry of reflections are well understood, analyzing reflections on arbitrary mirror surfaces is still complicated, and quantitative classification of reflection distortions still remains an open problem. Most research on analyzing reflections has focused on catadioptric mirrors, such as hyperbolic and parabolic mirrors to synthesize different pinhole cameras with a wide field of view [36, 2, 22, 8, 40], to obtain single viewpoint optics. 113 (b) (a) (c) Figure 8-1: A reflected image seen on a car window. Subregions illustrate multiperspective distortions such as shearing (yellow), uneven aspect ratio (red), and duplications (green). This setup requires accurate alignment of the viewing camera. Different methods have been proposed to calibrate these catadioptric imaging systems [18, 36]. When the camera moves off the focus, the reflected rays no longer converge to a single point. The envelop of these rays, often referred to as the caustics, quickly evolves into complicated shapes with discontinuities such as cusps [36, 37]. Closedform solutions of caustics are restricted to mirrors with simple shapes [36, 34]. The caustic surface models every ray as originating from a single, but spatially varying, pinhole camera, but it does not provide much insight into the group behavior of neighboring rays, and hence, cannot be used for interpreting the imaging properties at local regions of the reflected image. In this chapter, I use the GLCs to analyze reflections and model caustics on arbitrary mirrors. While most previous research has been restricted to simple parametric reflectors such as spherical or conical mirrors [8, 40], and equiangular mirrors [34], the GLC method applies robustly to reflections off of any parametric or mesh surface. 114 Ray Camera Figure 8-2: At each point on the surface, the reflected ray is mapped into the ray space by intersecting with the two parametrization planes. 8.2 8.2.1 GLC Reflection Analysis Multiperspective Reflection Model General reflected images can be analyzed using the GLC framework. Given a viewing camera and a mirror surface, we can map each reflected ray into [a, 7-, u, v] ray space. Assume the mirror surface is of form z(x, y). We can compute every reflected ray as follows: rxrl, r, r] = i - 2(A •-i)A (8.1) where i is the direction of the incident ray and hi is unit normal direction that can be computed as [-zx, -zy, 1]. Intersecting the ray with the st and uv plane as shown in 115 Figure 8-2, the [a, T, u, v] coordinate of the ray can be computed as: r r [,, u,v] = [,rz, rx rzy x-z rY (8.2) rz] All variables r, z, a, T, u, and v are functions in x and y, hence, the set of reflection rays from the mirror surface form a ray space parametric manifold in x and y E (x, y = u (x, y), r (x, y), u (x, y), v (x, y I (6.2) At every point on the reflected image manifold [a, r, u, v], a tangent plane can be computed with two spanning vectors d and d2 using equation 6.3 as: = d [ax, Tr,ux, vx] = [, d2 [[a and = where 0 =: , u, v = - z, x, 1- O, -zx -zx] (8.3) 1-zY - zy] ,-y] - zy . Its characteristic equation can be computed by choosing three points tangent plane, which are E(x, y), E(x, y) + d1 , and E(x, y) + d2 using equation 6.4: u + Aa v + AT 1 (u+u)+ (a+a) (v + v) + A( (u + uY)+ A( + Y) (v + v) + A( + -) + r-) 1 =0 1 yielding to the first characteristic equation AA2 + BA + C = 0 where A = aT uy - ayT B -= axVy- yVx - TxUy+ TyUx C = UxVy - UVx (6.5) The discriminant, A, can be computed as B 2 116 - 4AC as (6.4) Orthographic Camera Pinhole Camera Figure 8-3: (a) If viewed from an orthographic camera, the parametrization plane is perpendicular to the incident direction. (b) If viewed from a pinhole camera, the parametrization plane at each point on the surface is parallel to its tangent plane. A = (TuY- T7Uy + xvy - ovx)2 - 4(axry - uy7)(uy - uY~v) (6.6) The second characteristic equation (edge-parallel condition) used to distinguish pinholes from pencils, and twisted orthographic from orthographic cameras can be computed using equation 6.7 as: xvx - TxXU = 0, (au ayVy - TyUv = 0 - O a~,)(v - vy) - (rT - rT)(u. - Uy) = 0 (6.7) The goal here is to model local reflections as one of the GLC types. To simplify this analysis, it is desirable to properly choose the parametrization plane. The Differential Invariance Theorem described in Chapter 6.3 provides the freedom to reparameterize any tangent plane without changing the local GLC-type-specific behavior on the reflection manifold. In this section, I show how to choose appropriate parametrization planes to analyze reflections off of arbitrary mirror surfaces when viewed from orthographic and pinhole cameras. 117 8.2.2 Orthographic Viewing Conditions Under an orthographic viewing condition all incident rays to the reflective surface share a common direction. By choosing the two parametrization planes perpendicular to this viewing direction, we can orient the coordinate system so that all incident rays have direction (0, 0, 1) as shown in Figure 8-3(a), and parameterize the mirror surface as z(x, y) with respect to the uv plane. (Since the uv plane is parallel to the cartesian xy plane, we can also write z(x, y) as z(u, v). However, in order to distinguish the ray parametrization [u, v] from the mirror surface parametrization, we use z(x, y) to represent the mirror) At every point on the mirror, the [, T, u, v] coordinate of the reflection ray can be computed using equation 8.2 as: [a,,UV]= -,-,x--,y-_ 2zx 2zxz a ,2zy a a 1017 V 2zyzI a where a= z + z2- 1 (8.4) Notice, the denominator a is zero if and only if the reflected ray is parallel to the parametrization planes. If it happens, we can choose a different set of parametrization planes. The most relevant coefficientsand discriminant of the first characteristic equation for determining the GLC type can be computed using equation 8.3 and 6.5 as: (8.5) A = -K:43 /3 B = -2zA-2 A a2 (zx + YY) +(x -- 402 T4 ((Zo _ zyy)2 + 4z4 ) The second characteristic equation can be computed using equation 8.3 and 6.6 118 as: ¢v~ - Tzu~ - == - TrU auu ay~y-Ty ~ (a - ay)(v- vy) -(T - Tu)(u - ) = a22-~zY u (8.6) 2,3 -/ a ZXY (z zyy) where = + y2+1 K z+Z r= 2y 1-)Z (8.7) (zx + zY + 1) 2 According to Table 3-1, if the local tangent plane has A = 0, the corresponding GLC is either a pushbroom camera when B $ 0, or an orthographic/twisted orthographic cameras when B = 0. From equation 8.5, A = 0 if and only if K = 0, because cannot be zero. Furthermore, K represents the Gaussian curvature at the point [20]. Therefore, K is zero at parabolic points and planar points, i.e., ZxxZ (8.8) = Z2y First let us consider A = 0 and B = 0 case, which corresponds to orthographic and twisted-orthographic cameras. Substituting A = 0 into B in Equation 8.5, we have zx =Zyy (8.9) Thus, Equation 8.8 and 8.9 can be satisfied at the same time if and only if Z = z (8.10) = yy= 0 This occurs when the local surface is a locally planar (flat) mirror. Substituting Equation 8.10 into the edge-parallel equation 8.6, it is easy to verify that the edgeparallel equations are zero when z = y = z 119 = 0. Thus, local reflected rays cannot be a twisted orthographic camera. In general, when A = 0, the reflected rays at the parabolic points locally behave like a pushbroom camera. Koenderink [20] shows that specular reflections exhibit "duplications", i.e., multiple images of a single point, at the parabolic points. This "duplication" is in fact a common feature of pushbroom cameras, as shown in Chapter 5. Now let us consider cases for A when A 0. From equation 8.5, it is clear that A cannot take on negative values, therefore the tangent GLCs cannot be of the bilinear GLC-type. Thus, local reflected rays either behave like a cross-slit camera when A > 0 or a pinhole/pencil when A = 0. By equation 8.5, A = 0 occurs only when zx = 0, zXX = zy (8.11) Substituting Equation 8.11 into the edge-parallel equation, it is easy to verify that the edge-parallel equations are zero whenever zxy = 0 and z = z. Therefore, all solutions with A = 0 correspond to pinhole GLC camera types, and no pencil cameras are possible. Furthermore, solving equation 8.11 gives a paraboloid. A similar result has been shown in [2]. In general, reflections observed under orthographic viewing conditions can only be cross-slit, pushbroom, pinhole, or orthographic images. No bilinear (oblique), twisted orthographic, or pencil images can be observed from a single-surface reflection image seen by an orthographic camera. 8.2.3 Perspective Viewing Conditions The analysis of a reflected image seen by a pinhole camera is more complicated. Assume that the pinhole camera's center-of-projection is at the origin. The GLC differential invariant property allows us to choose a different parametrization for every reflected ray. One can choose the parametrization planes to be parallel to the tangent plane of the 3D reflection surface at each reflected ray, and the uv plane to pass through the origin, as shown in Figure 8-3(b). This transformation 120 is analogous to the Legendre transformation of the curve equation [3, 24]. The mirror surface is modelled as a Monge function z(x, y) with respect to the parametrization planes. Under this choice of parametrization, we must have z = z = 0 at the reflected ray we analyze. The direction [a, r, 1] of the reflection ray can be computed using 8.2 as x(z2 = 2 _ 1) + 2zxzyy- x Z; (8.12) ~~~~~~~~~~(8.12) 2zz -2zxx - 2zy -za T - 2zxzyx - y(z2 - z2 + 1)-2zz -2zxx - 2zyy -zo The A and B coefficients of the first characteristic equation can be computed using equation 6.5 and 8.12 as: A = (2x 2 (2z(zxzyy - z2y) + zxx) (8.13) +4zxyxy + 2y2 (2z(zxzZyy- z2y) + zyy) +z(4z 2 (zXzYY - zxy 2 ) + 2z(zxx + zyy) + 1))/z 3 B = -((2(x 2 (4z(zxxzyy- z) + 3zxx) +6zxyxy + y 2 (4z(zxxzyy - z2y) + 3zyy) +z(4z 2 (zxz~ - zx2) + 3z(zxx + zyy) + 2))))/z 2 The discriminant A of the first characteristic equation can be computed as: A= (4(zx2x 4 + 4z +2x 2 (y 2 (zxz~ + 2zx2) + z2 (z2 _ zZ +2zxy)) + 4zxyxy(z +2 Y4- z yX 3 y y 2 + z 2 (zx + zy)) 2Y2z 2 (z~zYY - 2zx - z2) +z4(z2x - 2zzyy + 4zxy + z;Y)))/z 4 (8.14) Finally the second characteristic equation (edge-parallel condition) can be com121 puted as: Xv - T u = 2(zx.xy + - Zxy(y2 + Z2 )) Z2 (8.15) -r = 2(zxyx 2 + zyyXy + z yz 2 ) 'y y ( - YY y- y )( ~~Z2 -- - v,) - (T - Ty (X2x_ -zxy) +Y 2 (Zx )(U - uY) + XY(ZXX_ zYY) Z Y) + Z2 (Zx _ z))/Z2 Next, we show A > 0. Notice A is quadratic in zy, therefore, we can treat zy as as the root of the quadratic equation A = 0, an unknown variable and solve for z () and if the solution for zxy has an imaginary part, it means for all zy, A is always > 0 or < 0. Solving A = 0 for zy, we get 816 -xy(zxX2 + zy y2 + z2 (zx + Z)) 2 2 2 2 (2(X + z )(y + z )) )lzxxx2 - zyyy2 )+ + Z2(Zx - zYy) z(2 +2 + (2(X2 + z 2)(y 2 + Z2)) Since z has an imaginary part, it indicates A > 0 or A < 0 for all zy. determine which case it is, we can simply substitute zy = 0 and z To = zyy = 1 into A, and it gives A = 4(x 2 + y2 ) 2 > 0. Therefore, we must have A > 0 for all zy. Thus, the tangent GLC cannot be a bilinear camera on a reflection manifold. Next, we show the tangent GLC cannot be a pencil camera. Consider A = 0 if and only if the imaginary part of zxY is zero and zy is the real part of Equation 8.16, i.e., zX(x2 + 2 zyy(y2 + z2 ) = )- -_ Xy(ZxX ZX"=2)2+Z Substituting z 2 2 0 (8.17) 2 + zyyY + z (zX + Z )) 2(X2 + z2 )(y 2 + z) and zy into the second characteristic equation (edge-parallel condition) 8.15, we obtain that second characteristic equations are zero. Thus, the edge 122 parallel condition is satisfied whenever A = 0. Therefore, no local reflections can be pencil-type GLC camera. Furthermore, the solution to Equation 8.17 for z can only corresponds to parabolic, hyperbolic, ellipsoid, and planar shapes, as shown in [36]. Next, we consider A = 0 and B = 0. Notice, notice both A and B are quadratic in zXXand zyy, we can solve A = 0 and B = 0 for zxx and zyy as 2zxyx + z 2 2(x + z 2 ) 12zxyx2 z- xy + 2zxyy 2 z + 2zy Z31 2(X2 + y2 + Z2)i(x2+ z 2) 2zyxy + z yy - 2(y 2 + z2 ) 12zyx 2 z- xy + 2zxyy 2z + 2zyz31 2(X Since both z and z 2 TZ i 1 + y2 + z2)1(y2 + z2) (8.18) ~~~~~~~~~~~(8.18) have imaginary parts, a real mirror surface exists for A = 0 and B = 0 if and only if the imaginary parts of Equation 8.18 are zero, i.e., 2z yx2 z - xy + 2zxy 2 z + 2zxyz3 And z and z = 0 (8.19) will correspond to real values as: 2zyxy + z 2(X 2 2 + (8.20) ) 2zxyxy+z 2 (y 2 + z 2 ) Solving equation system 8.19 and 8.20 for for z, zy and zyy gives y2 + z2 X = 2z(X 2 + y2 + Z2) xy Zay= 2z(x 2 + y2 + 2 z = 2) 2 X 2 + Z2 2 + y2 + -2z(x (8.21) 2) Substituting Equation 8.21 into the second characteristic equation, we obtain all 123 three the edge parallel equations are also zero, thus, there is no twisted orthographic cameras when A = 0 and B = 0. Unlike the orthographic viewing case, in which we choose a fixed 2PP, here we change the parametrization plane according to the normal of the mirror surface. Thus, recovering the surface requires an inverse Legendre transformation. Oren and Nayar have showed in [24]that local mirror surface must correspond to planar/hyperbola/ellipoid in order for the view point to reflect onto another point (in our case when A = 0), and the mirror surface must correspond to a parabola when the view point is reflected to parallel rays (in our case when A = B = 0). In general, all reflections observed by a pinhole or an orthographic camera can be characterized by only four types of GLCs: cross-slit, pushbroom, pinhole, or orthographic. No pencil, twisted orthographic, or bilinear cameras are observed. The non-existence of these three types of GLCs is due to the incident ray structure and the differential geometry property of the mirror surface. Notice these three cameras contain purely twisting rays. However, there is no twisting rays from an observing pinhole or orthographic camera. Furthermore, the mirror surface satisfies differential geometry properties. For instance, at each point on the surface, there always exist two principal directions, and along the principal directions, the surface normal do not twist [20]. Thus, the incident rays do not twist and the mirror surface normal cannot be purely twisting. Therefore, the reflection rays within a neighborhood cannot be purely twisting, and local reflection GLCs cannot be pencil, bilinear, and twisted orthographic when viewed from a pinhole or orthographic camera. Furthermore, the GLC analysis only applies to local reflections. It does not show if there exist such mirrors that can map all reflected rays to pass through a common cross-slit, pinhole, orthographic, or pushbroom camera. Oren and Nayar have shown there does exist quadratic mirrors that reflect all rays from a pinhole/orthographic camera to pass through a different pinhole/orthographic camera. In Appendix A, I show there does not exist such mirrors that reflect all rays from a perspective viewing camera to pass through a common cross-slit. 124 8.3 Caustic Surfaces The caustic surface is the envelop of the reflected rays. Near the caustics, the reflected rays bunch up together. Previous approaches [24, 36, 35] modelled caustics as the loci of the virtual viewpoints of the viewing pinhole camera with respect to the mirror surface. However, there are two fundamental problems to treat the caustics as the loci of the virtual pinholes. First, the reflected rays form a pair of caustics rather than an single caustic surface in general [7, 36, 35]. The virtual pinhole camera model, therefore, is unable to explain why the single viewpoint is mapped to two different virtual viewpoints. Second, the virtual pinhole model does not provide much insight into the group behavior of neighboring reflection rays, and hence, cannot be used for interpreting the image distortions in reflections. In this section I will show the relationship of the local GLC model a reflection surface point to the caustic surface of the reflection. I first compute the caustic surface using the standard Jacobian method [7, 36]. Consider each reflected ray is parameterized as [, r, u, v] with source S(u, v, 0) and direction V(a, r, 1). The Jacobian matrix det[J(S + AV)] can be computed as: u + a v. + ATx a u + AVy v + Ay T 0 0 1 =0 (8.22) which corresponds to the same first characteristic equation of the local GLC model as equation 6.4. Therefore, the unions of the solutions to the characteristic equation for every local GLC form the caustic surface. Since the roots of the first characteristic equation often correspond to cross-slit cameras, which have two solutions, caustics surfaces are formed in pairs. A similar result is shown in [37]. For rotationally symmetric mirrors viewed along its rotation axis, the rotation axis is one of the caustics. 125 8.3.1 Cross-slit Rulings However, the first characteristic equation not only gives two caustic points, but also two slits, i.e., points with directions. In the conventional caustic analysis method, the caustic surface is regarded as the loci of a virtual viewpoint for every single ray. Our approach models the group behavior of nearby rays as a local GLC where rays simultaneously pass through the two slits, one on each of the caustic surfaces. Therefore, the obtained slits not only provide the caustic surface but also reveal a ruling upon it. Specifically, the local GLC slits are always tangent to the caustic surface, as proved in the following theorem. Theorem 8.3-1. At any ray r on a continuous ray manifold, if the local GLC model at r is a cross-slit, then the two slits are tangent to the caustic surfaces of the ray manifold at r. Proof: I show that the two slits lie on the tangent plane of the caustic surface at r[a, r, u, v]. Using the Jacobian method, the caustic surface has form Caus [u, v, 0] + A[a, T, 1], thus the two tangent directions c 1 = and c72 of the caustic surface at r can be computed as Cs = Caus, = [ux + Aa + -aOX, v. + AT + Ar, A] (8.23) cs2 = Causy = [uy + Aya + Say, vy + Ayr + Ary, Ay] Next, I compute the direction of each slit. Recall that the first characteristic equation of the local GLC is computed by picking three rays r = r, r2 = r + rx and r3 = r7+ ry so that all three ray pass through some slit at depth A. Therefore, choosing the three points P1 , P2 and P3 along rl, r 2, and r3 as P1 = [u + Aa, v + A, A] (8.24) P2 = [(u + u) +A( + ),(v+vx) + (T +), A] P3 = [(u+uy)+A( + 126 ),(v+vy)+ ( + ), A] P1 , P 2 , and P 3 must lie on a line. Without loss of generality, suppose P1 and P2 do not coincide, the direction of the slit can be computed as d= P 2 -P 1 = [u + a, v + A, O] (8.25) Finally, I show slit P2 P1 lies on the tangent plane of the caustic surface. To do so, I show d, sl, and C62 are linearly dependent. This is because Ux+ Ax Det[c'S, 's, d]= v + Ax 0 u + Axo + Ax vx + AzT + ATx A uY+A y a+A+A Vy+AYT+ATry Ay y =0 (8.26) Therefore, the slit lies on the tangent plane of the caustic surface and hence it is tangent to the caustic surface. If P ad P2 coincide, then we can pick P and P3 to compute dand similar results hold. In Figure 8-4, I illustrate three commonly used mirror examples. The mirror surface is drawn in cyan, the caustic surface in blue and viewpoint as blue circle. Notice, the rotation axis is the second caustic surface for both hyperbolic and spherical mirrors. The slits are shown in blue over the caustic surface in cyan. These rulings determine which rays lie on the local tangent GLC and the local distortions seen in the reflection, as I will show in Chapter 8.4. In addition, the spatial separation of the corresponding slits are also important. If the two slits intersect, the local GLC corresponds to a pinhole camera. Notice, the center of the hyperbolic and parabolic mirrors satisfy this condition, therefore the center of the image looks the least distorted. In the cylindrical mirror, however, the two slits at the cusp do not intersect, which results in the apparent bending as seen at the center of the image in Figure 8-5c. The rulings of the slits on the caustic surface are closely related to the fact that the 3D caustic surface exists only if the reflected rays form a developable surface which is not a generalized cylinder or a cone [6, 24]. This indicates that although 127 A1\ kI ku) Figure 8-4: The caustic surface and its ruling of typical mirrors. The viewpoint is shown as a blue sphere. (a) A hyperbolic mirror. (c) A parabolic mirror. (e) A cylindrical mirror. (b), (d), and (f) show the ruling of the caustics of (a), (c), (e), respectively. 128 (a) (b) (d) (e) Figure 8-5: Aspect ratio distortion. Top row: reflected images of a hyperbolic, spherical, and cylinder mirror. Bottom row: estimated ratio distortion by the slits distance of the respective mirrors. local reflections are all cross-slits in general, there does not exist a mirror that maps all rays to pass through a single cross-slit with an exception in the case of pinhole. In Appendix A, I show an alternative proof using differential geometry. 8.4 Characteristic GLC Distortions The GLC analysis has shown that reflections viewed from a pinhole or an orthographic camera off any mirror surface can only locally behave as one of the four GLC cameras. The distortions seen in the reflected image can be characterized by analyzing the imaging property of these cameras and their collineations onto the mirror surface. To simplify the analysis, we can choose the parametrization plane to be parallel to the tangent plane at the mirror surface point we analyze, so that the collineation degenerates to a simple linear transformation. 129 8.4.1 Cross-slit Distortion The image distortions seen by a cross-slit camera depend on the spatial relationships of the two slits and the image plane [47]. Specifically, if one slit is much closer to the image plane than the other, the orientation of the image will be dominated by the corresponding slit. In Figure 8-5, I compare the ray-traced reflection images seen from hyperbolic, parabolic and cylindrical mirrors. The ruling of the closer caustic surface is shown in Figure 8-4. It determines the orientation of maximal stretching in the reflected image (e.g., the image of the hyperbolic and parabolic mirrors form concentric circular patterns and the cylinder mirror exhibits vertical and bending patterns). The distance between the two slits determines the aspect ratio of orientation distortions. In fact, pushbroom, orthographic, and pinhole cameras can all be characterized as special cases of cross-slit cameras [47]. When the two slits intersect, it transforms into a pinhole camera with a small aspect-ratio distortion. When one of the slits goes to infinity, the cross-slit transforms into a pushbroom camera with large aspect-ratio distortions. In Figure 8-5, I plot the slit-distance image of the three mirrors. The hyperbolic mirror has an overall small distance map (5d), thus, its image (5a) has a relatively more uniform aspect-ratio distortion. The spherical mirror has small distance map near the center, that grows larger radially away from the center (5e). Thus, we observe more severe geometric distortions near the boundary of the mirror than at its center (5b). The cylindrical mirror has an overall large distance map (5f), and the whole image (5c) is stretched vertically since the vertical slit is generally closer to the image plane. When the distance between the two slits is infinitely large, the local GLC corresponds to a pushbroom, and things faraway will map to duplicated images, resulting in a singular event, as shown in Figure 8-1(b) and 8-6. These singular events often appear at the parabolic points when viewedfrom faraway (approximately orthographically), as I have shown in Chapter 8.4. A similar result is shown in Koenderink's [20] analysis of specular reflections. 130 8.4.2 Predicting Distortions on Mesh Surfaces The reflections seen on a general curved surface are difficult to analyze using conventional methods, because closed-form solutions are difficult to obtain. The local GLC reflecion analysis framework, based on parametric surfaces, can easily be extended to handle general reflection surfaces . The procedure involves representing the surface using a piecewise planar triangulation, with an actual surface normal at each vertex of the mesh. For each triangle of the mesh, we can compute the three reflected rays using the vertex normals, and model it as a GLC. To minimize errors, I choose the average reflected direction of the rays as the normal direction of the parametrization plane. By computing the characteristic equation and two slits of the GLC for each triangle, we can measure the slit distances from the surfaces and estimate the aspect-ratio and duplication distortions within each triangle's interior. In Figure 8-6, I give an example of the pear-shaped mirror modelled by the following function z = exp(_ x 2 + y2 ) )_ ( x 2 2 + y2 ) The surface is sampled into a triangle mesh, as shown in Figure 8-6(a). (8.27) At each triangle, I compute the local GLC and show the distance map between the two slits in Figure 8-6(b). The brightest region predicts where the pushbroom distortions occur. Figure 8-6(c) shows a ray traced image of the original analytic surface. The GLC-basecl method predicts the low and high distortions regions, as shown in the yellow, red, and blue rectangle overlays. In fact, the two bright bends estimated from the mesh surface correspond to the parabolic curve (red-dashed curve) of the surface. Duplicated reflections can be seen on either side of these parabolic curves, as shown in the red and blue outlined regions. In summary, I have presented a novel method to analyze reflections on arbitrary surfaces using a specific linear class of multiperspective cameras. I provided an analytical framework to locally model reflections as specific multiperspective cameras around every surface point based on the theory of general linear cameras. The GLC framework better characterizes the image distortions seen on irregular mirror surfaces 131 (a) *L~uf~f; 1~-~-~~ :~4": (c) Figure 8-6: Estimating reflection distortions of a general surface. (a) A pear-shaped mesh. (b) Estimated distortions. Bright colors correspond to large distortions (closer to pushbroom) and dark colors correspond to small distortions (closer to pinhole). (c) A ray-traced image of the parametric surface. Notice reflections at regions around the red dashed line correspond to brightest regions in (b). as well as the conventional catadioptric mirrors than do classical methods, which model the local neighborhoods of reflected rays as a pinhole approximation. The local multiperspective model also reveals important relationships between the two caustic surfaces and reveals an important surface ruling of the caustics. Finally, the GLC modelling approach provides a new tool for designing special featured mirrors or to avoid undesirable geometric image distortions. 132 Chapter 9 Conclusions and Future Work In this thesis, I have presented a new analytic camera model called the General Lin- ear Camera (GLC) model. The GLC model unifies many previous multiperspective camera models as well as the pinhole camera into a single representation. The GLC model is both general and linear in the sense that, given any vector space where rays are represented as points, it describes all 2D affine subspaces (planes) formed by the affine combination of 3 rays. I developed theories of projection and collineation for the GLC model and used these theories to explain various multiperspective distortions. I further showed that the GLC model provides an intuitive physical interpretation and is closely related to the caustics of the rays. Finally, I demonstrated how to use the GLC model for applications in computer vision, computer graphics, and optical design. In particular, I have shown how to use GLCs for modelling and rendering multiperspective images and characterizing real multiperspective imaging systems such as catadioptric mirrors. 9.1 The Theory of GLC The key insight underlying the GLC model is that any imaging system that captures a smoothly varying set of rays can be mapped to a 2D manifold in a 4D ray space. The characteristics of this imaging system, such as its projection, collineation, and image distortions can be analyzed by studying this ray manifold. The 2-D tangent 133 planes of this manifold correspond to the General Linear Camera model defined here. In addition, I have analyzed many important algebraic and geometric properties of these fundamental linear subspaces. Each GLC describes a specific set of rays set of rays. These ray sets define an elemental set of multiperspective camera types. The GLC model also unifies many previous perspective and multiperspective cameras, including pushbroom, cross-slit, linear oblique cameras, within a single framework. It also proposes three new, previously unexplored multiperspective camera types. I derived a set of characteristic equations for identifying 8 atomic GLC types, and proved that these types are characterize all possible linear multiperspective camera models. I also introduced the GLC projection and collineation theory in Chapter 5 that characterizes the image distortions commonly observed in multiperspective imaging systems such as seen in curved mirrors. A significant difference between the GLC approach and the traditional multiperspective analysis is that, the traditional approaches model the behavior of the neighboring rays (within a small area on the image) as spatially varying pinholes while the GLC framework models it as specific multiperspective cameras. I have shown in Chapter 4 that the local pinhole model does not provide much insight into the group behavior of neighboring rays, and hence, cannot be used for interpreting the imaging properties (multiperspective distortions) at local regions. The local GLC model, however, models the group behavior of the neighboring rays and it explains and predicts the multiperspective distortions. In addition, I showed the close relationship between the local GLC model seen at any reflected ray and the caustic surfaces of reflections. In particular, I have shown that the caustic surfaces of the rays, if they exists, correspond to the loci of the local cross-slit GLCs. The two slits of the local GLC model not only provide the geometry (depth) of the caustics, but they also provide a ruling over the caustics. These rulings are particularly important for characterizing the multiperspective distortions of local image regions. Finally, I have proved that the GLC-type of any infinitesimal surface patch is 134 invariant to the choice of parametrization plane, and therefore, intrinsic to the shape. The limitation of the GLC framework is the use of two-plane-parametrization. The algebraic representation of the ray (and hence the ray manifold) changes when the parametrization plane changes. This inconsistency has limited the usage of 2PP for the effective analysis of the ray space. The Differential Invariance Theorem partially overcomes this problem by showing the local GLC type is invariant to the parametrization plane. However,the actual GLCs may still change when the parametrization plane changes. For instance, a local cross-slit GLC remains as a cross-slit when using a new parametrization plane not parallel to the old one. However, under the new parametrization, the two slits in the local GLC must be different from the old parametrization. This is because the two slits are parallel to their parametrization planes. Therefore, the GLC analysis is still parametrization-dependent. Ideally, a coordinate-free ray representation and analysis can completely resolve this problem by representing the local GLC model as the differential geometry property of the rays. In the future, I plan to extend this thesis to develop a ray space differential geometry framework. 9.2 GLC Applications I have also demonstrated applications of the GLC model in the fields of computer vision, computer graphics, and optical design. GLCs c(,an be used for the direct rendering of multiperspective images. I showed that GLCs, when constrained by an appropriate set of rules, can be laid out to tile the image plane and, hence, generate arbitrary multiperspective renderings. I provided a complete of set of rules for tiling an image plane with GLC fragments to achieve consistent rendering. I used the GLC framework to render a broad class of multiperspective images, such as multiperspective panoramas, neocubist style renderings, and faux-animations from still-life scenes. GLCs also provide an analysis framework for studying reflection ray manifolds via local GLC models. I proved that local reflections observed on arbitrary mirrors can 135 originate from only four of the eight possible GLC cameras: pinhole, orthographic, pushbroom, and cross-slits. I characterized the distortions seen in reflections according to the local GLC model and collineation and connected specific distortions to the mirror geometry. Finally, I showed how to use the GLC analysis to assist in-mirror design and predict distortions seen in catadioptric imaging systems. An arbitrary mirror surface can be approximated as a triangulation with surface normals at each vertex. The three reflected rays associated with the vertices of a triangle form a GLC. One can then find an optimal parametrization plane for the GLC and compute the local crossslit using the GLC characteristic equations. Finally, the distortions can be estimated by the local cross-slit model using the GLC projection, collineation, and caustics analysis. 9.3 Future Work There are several directions for future research related to the GLC model. GPU-based GLC Rendering: Most GLC images shown in this dissertation are synthesized from either precaptured or rendered light fields. In the future, commodity graphics hardware might be used to directly render GLC images directly from the scene geometry. The GLC projections of a line onto the GLC image plane is a quartic rational. Thus, it is possible to modify the rasterization unit or the shader to efficientlyrender the geometry directly into a GLC image using the graphics hardware. A real-time renderer will be beneficial to the interactive design of multiperspective rendering, the creation of backdrops for cel-animation, and image-based animations. GLC-based Reflection Model: The GLC reflection analysis shows that a reflected image is a 2D manifold of rays that can be indexed into the 4D light field ray space. Based on this concept, we have developed new rendering algorithms using graphics hardware to achieve real-time renderings of accurate reflections by replacing the 2D environment map with six 4D light field slabs to represent the surrounding scene [44]. Each reflected ray is rendered by indexing into these pre-captured 136 environment light fields and accurate reflections are achieved with motion parallax at interactive frame rates. The GLC framework can also be extended in two other directions. First, the GLC reflection model could be combined with image-based rendering techniques. In particular., it is interesting to design specially curved mirrors to efficiently capture light fields as well as mirrored surfaces for autostereo 3D displays. The GLC reflection analysis has shown that local reflections can only originate from four types of GLCs: pinhole, orthographic, pushbroom, and cross-slits. In fact, all four GLCs are specially cases of the cross-slits. Thus, using special mirrors, it is possible to more efficiently sample the ray space via a different set of 2D subspaces. Second, one could render reflections directly from GLCs without using light fields. This requires developing effective algorithms to discretize the reflection manifold in terms of GLCs (especially cross-slits) and to adapt the conventional graphics pipeline for rasterizing GLC images. I am also interested in designing actual GLC cameras and new stereo algorithms to enable 3D reconstructions from very wide field-of-view images, as well as other multiperspective images. Most of the GLC models satisfy Seitz's criterion and are suitable for binocular stereo vision. Combined with the reflection analysis, it is possible to design new types of mirrors, or even use arbitrary mirrors as novel stereo imaging systems. Furthermore, the GLC reflection analysis shows how to locally map a single viewpoint to a cross-slit camera. If we replace the view point with a light source, it is possible to transform a point light source to a cross-slit light source. Many image- based relighting approaches are restricted by the geometric constraints of the light source, and by designing a different type of lighting condition, it is possible to improve the way for effectivelymeasuring and sampling the radiance off the surface and, therefore, benefit applications such as measuring the surface BRDF. Caustic-driven Differential Geometry: Finally since the GLC model provides a first-order differential analysis of any ray manifold, it has the potential for directly analyzing 3D geometry. In particular, if we model the surface normals on a surface 137 as rays, similar to the caustics of reflections, the local GLC model can effectively approximate the focal envelops of the surface. The loci of the GLC cameras form the focal surface of the 3D geometry. Many previous researchers have shown that the focal surface embeds important differential geometric quantities such as the principal curvatures and principal directions of the surface. Thus, the local GLC model, which not only gives the depth of the focal surface but also the ruling of the focal surface might provide a novel strategy for approximating the focal surface and computing the surface curvatures. 138 Appendix A Non-existence of Cross-slit Mirrors In Chapter 8, I have shown that local reflections can only correspond to four GLCs under the perspective viewing conditions. They are: pinhole, orthographic, pushbroom, and cross-slits. The first three types of cameras are special (degenerate) cases of the cross-slits. However, this analysis is local and only characterizes the reflected rays within an infinitesimal patch. I did not show whether there exist a mirror that can reflects all rays from a viewing pinhole camera to a common cross-slit camera. Oren and Nayar have shown that it is possible to use hyperbolic, ellipsoid, and parabolic mirrors to reflect all rays from a perspective viewing camera to pass through a different (virtual) perspective or orthographic camera, which are degenerate cases of the cross-slits. In this section, I show there does not exist a mirror that reflects all rays from a perspective viewing camera to a common non-degenerate cross-slit camera. Lemma A.0-1. (Ellipsoid) Suppose S is a surface, which reflects all rays passing through p to rays all passing through q. If q is at finite, S must either be an ellipsoid or hyperbola,and p, q are the focal points of S. If q is at infinity, then S is a parabola. Proof: Consider any point r E S, since ray pr is reflected to qr, we must have r Pp r qi dr = Jr -p Jr-q'dr 139 (A.1) Figure A-1: Reflection between pinhole cameras. Left: an ellipsoid mirror reflects between two pinhole cameras on the same side. Right: a hyperbolic mirror reflects between two pinhole cameras on different sides. as is shown in Figure A-1. This can be reformulated as (r - p) . dr ± (r - q) dr Ir - pi Ir - qI Hence we must have Ir - pl dr - p 2 Ir- pi dr - q 2 Ir - q1 d - Plt dr - q = 0 (A.2) Ir - ql = const, therefore, S must be an ellipsoid or hyperbola. If q is at infinity, we then use q to denote the direction of itself. It is easy to show that r-p q 0 ir- pl Iqj dr = (A.3) Ir - pI + r -q = const (A.4) (r- + Hence And therefore, S is a parabola. The lemma above shows the necessary and sufficient condition to mirror one pinhole camera to another pinhole or an orthographic. These mirrors are well known as parabolic mirrors or ellipsoid mirrors, depending on whether the two points are on the same side of the mirror. A Pinhole camera can also be mirrored to an orthographic mirror using a parabolic mirror. Theorem A.0-2. (Cross-slit Viewpoint Constraint) Given two lines 11, 12 in general position, and a viewpoint q, denote plane 7rl = span{ll, q} and plane r2 = 140 -If"i~' nrs I ; 1~# 4,Cl'r: 4,' 4,:- Figure A-2: Geometric constraints on mapping between pinhole and cross-slit camera. span{12, q}, if the mirror surface S exists that maps all rays passing through q to pass through both 11 and 12, then we must have 7r1 I r2 . Proof: Notice the intersection of 7rl and r2 corresponds to the ray r that passes through q and lI and 12, as shown in A-2. Suppose irl n l = pi and 7r2 f 2 consider S 7r = cl, S n 7 2 = = p2. And c2 , and S n r = po. Obviously, po E cl n c2 . Now consider all rays r2P 1 on 7r2 , r2 E C2 . Notice p 1r 2 lie on 7r2 and thus, it must intersect 12 and must be reflected to r2q. This means, the normal of S at r2 must be on P2 as well and coincide with the normal of curve c2 . And because c2 is planar, c2 must be the principal curvature line on S. Same case hold for cl. In addition, it is easy to see both cl and c2 are either ellipse or a hyperbola from Lemma A.0.1. Finally consider ray pipO is reflected to P2po, hence pip2 is perpendicular to the surface at po. Notice at po, cl and c2 are principle curvature lines, hence we must have 7r1 .L 7r2 . Theorem A.0-3. (Nonexistence) There exist no surface, which reflects all rays in an cross-slit camera to a pinhole camera. Proof: We use the same notation as in Theorem A.0.2. First we assume pl, p2 141 and q are on the same side of S. We consider the two ellipse can be written as: cl: r-p21 + jr-ql = C1 C2: r- Pl + r -ql = C2 Now denote all points on 11 as p(t) and all points on 12 as p(s). span{p(t),12}, ellipse c(t) = S (A.5) (A.6) Plane r(t) = 7r(t). Notice, because 12 is on r(t), we therefore can talk about all rays passing through p(t) on pi(t). Suppose 7r(t)n cl = b, then p(t), P2 and b must be on the same line, and b E c(t), hence, on c(t), all rays passing through p(t) must be reflected to pass through q. We have: c(t): r-p(t)j+lr-ql = b-p(t)l+lb-ql = b-p 2 J+Jb-ql+Jp 2 -p(t)l = Cl+Ip(t)-p21 (A.7) Similarly, we have the formula for ellipse c(s) c(s): r - p(s)l + r - ql - C2 + p(s) - Pll (A.8) Thus, the surface S can be treated as woven by two family of ellipses c(t) and c(s), by construction r(s, t) = c(t) n c(s) is on the ray p(t)p(s). We, therefore, get the following equations system for r(s, t) as follows: Ap(t)+ (1 - A)p(s) = r(s, t) Ir(s,t) - p(t)j + jr(s, t) - q jr(s,t)-p(s) + Ir(s,t)-q = C1 + Ip(t)- P21 = C2+ jp(s)-Plj (A.9) (A.10) (A.11) This equation system is over determined. We want to show that it has no solution for any open set of (s, t). To do so, we replace r with Equation A.9 into A.10 and A.11, and we get 1 - Allp(t) - p(s)I + jr - q = p(t) - P21+ C1 142 (A.12) AllIp(t)- p(s)I + r - q = p(s) - pl + C2 (A.13) When A < 0 or A > 1, the difference of the above two equations give us Ip(t) - p(s)I + p(s) - PlI - (p(t) - P21= C1 - C2 (A.14) Notice p(t) and p(s) are two lines in 3D space and it is not possible for the left hand side of the equation above to be constant. If A is between 0 and 1. Then the difference between (21) and (22) is (1 - 2A)Jp(t) - p(s)I = I(p(t) - P21- Ip(s)- PlI + C 1 - C2 (A. 15) Solving for A, we obtain 1 Jp(t)- P2 - p(s) - P + C - C2 A=-2 2Ip(t) - p(s)I (A.16) And we can solve r by equation A.9 as -P21 -P(S) p(t) - P r = p(t) + p(s) + (p(s) - p(t)) + C1 - C2 (A.17) ]pMt - p(s) I The summation of A.12 and A.13 is: J(p(t) -p(s)I + 2jr - qJ- p(t) -P21 - Jp(s) -pl = C1 + C2 (A.18) Given p(t) and p(s) are points on two different lines, this equation cannot be further simplified. Therefore, there is no way that the left hand side is a function of (s, t) while the right hand side is a constant. In general, we can assume S is formed by two family of curves, either ellipse or 143 hyperbolas. And the surface is formed by three constraints as: Ap(t) + (1 - A)p(s) = r(s,t) Ir(s,t)-p(t)l ± jr(s,t)-q (A.19) = C1+ Ip(t)-P21 (A.20) Ir(s,t)-p(s)l ± jr(s,t) - q = C2+ jp(s)-Plj (A.21) For a small open set of (s, t), the signs are fixed. Similarly, we can show that that does not exist such mirror surfaces. 144 Bibliography [1] M. AGRAWALA, D. ZORIN, T. MUNZNER: Artistic multiprojection rendering. In Eurographics Rendering Workshop 2000 (2000). [2] S. Baker and S. K. Nayar, "A Theory of Single-Viewpoint Catadioptric Image Formation," International Journal on Computer Vision, 35(2):1-22, 1999. [3] C. Bellver-Cebreros and M. Rodraguez-DAnta, "Caustics and the Legendre Transformation," Optics Communications, 92(4-6):187-192, 1992. [4] R.C. Bolles, H. H. Baker, and D. H. Marimont: Epipolar-Plane Image Analy- sis: An Approach to Determining Structure from Motion. International Computer Journal of Vision, Vol. 1 (1987). [5] M. Born and E. Wolf, "Principles of Optics," Permnagon Press, 1965. [6] J. Bruce and P. J. Giblin, and C. G. Gibson, "Curves and Singularities," Cambridge University Press, second edition, 1992. [7] D. G. Burkhard and D. L. Shealy, "Flux Density for Ray Propagation in Geo- metrical Optics," Journal of the Optical Society of America, 63(3):299-304, March 1973. [8] J.Chahl and M. Srinivasan, "Reflective Surfaces for Panaromic Imaging," Applied Optics, 36(31):8275-8285, 1997. [9] F. DURAND: An invitation to discuss computer depiction. In Symposium on Non-Photorealistic Animation and Rendering (NPAR) (2002). 145 [10] A. GLASSNER: Cubism and cameras: Free-form optics for computer graphics. Microsoft Research MSR-TR-2000-05 (2000). [11] S. Gortler, R. Grzeszczuk, R. Szeliski, and M. Cohen: The Lumigraph. Proc. ACM SIGGRAPH '96 (1996) 43-54. [12] X. Gu, S.J. Gortler, and M.F. Cohen. Polyhedral geometry and the two-plane parameterization. Eurographics Rendering Workshop 1997 (1997) pages 1-12. [13] R. Gupta and R.I. Hartley: Linear Pushbroom Cameras. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 9 (1997) 963-975. [14] A.J. HANSON AND E.A. WERNERT: Image-based rendering with occlusionsvia cubist images. In IEEE Visualization '98 (1998), Ebert D., Hagen H.,, Rushmeier H., (Eds.), pp. 327-334. [15] R.I. H-artley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2000. [16] H-Y. SHUM AND L-W. HE: Rendering with concentric mosaics. In Proc. ACM SIGGRAPH '99 (1999), Computer Graphics Proceedings, Annual Conference Series, ACM, ACM Press / ACM SIGGRAPH, pp. 299-306. [17] A. ISAKSEN, L. MCMILLAN, AND S. GORTLER: Dynamically reparametrized light fields. In Proc. ACM SIGGRAPH '00 (2000), Computer Graphics Proceedings, Annual Conference Series, ACM, ACM Press / ACM SIGGRAPH, pp. 297306. [18] S. B. Kang, "Catadioptric Self-calibration," In Proc. CVPR, I:201-207, June 2000. [19] R. Kingslake, Optics in Photography. SPIE Optical Eng., Press, 1992. [20] J.J. Koenderink, "Solid Shape," MIT Press, 1990. [21] M. Levoy and P. Hanrahan: Light Field Rendering. Proc. ACM SIGGRAPH '96 (1996) 31-42. 146 [22] S. K. Nayar, "Catadioptric Omnidirectional Cameras," In Proc. CVPR, pages 482-488, 1997. [23] B. Newhall, The History of Photography, from 1839 to the Present Day. The Museum of Modern Art (1964) 162. [24] M. Oren and S. K. Nayar, "A Theory of Specular Surface Geometry," Int. Journal on Conmpute Vision, 24:105-124, 1996. [25] T. Pajdla: Stereo with Oblique Cameras. Int'l J. Computer Vision, vol. 47, nos. 1/2/3 (2002) 161-170. [26] T. Pajdla: Geometry of Two-Slit Camera. Research Report CTU-CMP-200202, March 2002. [27] S. Peleg, M. Ben-Ezra, and Y. Pritch: Omnistereo: Panoramic Stereo Imaging. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 3 (2001) 279290. [28] P.Rademacher and G.Bishop: Multiple-center-of-Projection Images. Proc. ACM SIGGRAPH '98 (1998)199-206. [29] A. Romn, G. Garg, and M. Levoy: Interactive Design of Multi-Perspective Im- ages for Visualizing Urban Landscapes. Proc. IEEE Visualization 2004. [30] S.M. Seitz: The Space of All Stereo Images. Proc. Int'l Conf. Computer Vision '01, vol. I (2001) 26-33. [31] J. Semple and G. Kneebone: Algebraic Projective Geometry. Oxford: Clarendon Press, 1998. [32] H.-Y. Shum, A. Kalai, and S. M. Seitz: Omnivergent stereo. In Proc. 7th Int. Conf. on Computer Vision (1999) 22-29. [33] D. Sommerville, Analytical Geometry of Three Dimensions. Cambridge Univer- sity Press, 1959. 147 [34] W. Stiirzl, H. J. Dahmen, and H. Mallot, "The Quality of Catadioptric ImagingApplication to Omindirectioinal Stereo," Proc. 8th European. Conf. on Computer Vision (ECCV 2004), 614-627. [35] R. Swaminathan and S. K. Nayar, "A Perspective on Distortions," In Proc. IEEE Computer Vision and Pattern Recognition (CVPR), Wisconsin, June 2003. [36] R. Swaminathan, M. D. Grossberg, and S.K. Nayar, "Caustics of Catadioptric Cameras," Proc. of IEEE International Conference on Computer Vision, Vancouver, Canada, July 2001. [37] R. Swaminathan, M. D. Grossberg, and S. K. Nayar, "Non-Single Viewpoint Catadioptric Cameras: Geometry and Analysis," Technical Report CUCS-004-01, Dept. of Computer Science, Columbia University, 2001. [38] T. Takahashi, H. Kawasaki, K. Ikeuchi, and M. Sakauchi: Arbitrary View Posi- tion and Direction Rendering for Large-Scale Scenes. Proc. IEEE Conf. Computer Vision and Pattern Recognition (2000) 296-303. [39] D. Wood, A. Finkelstein, J. Hughes, C. Thayer, and D. Salesin: Multiperspective Panoramas for Cel Animation. Proc. ACM SIGGRAPH '97 (1997) 243-250. [40] Y. Yagi, S. Kawato, and S. Tsuji, "Real-time Omnidirectional Image Sensor (copis) for Vision-Guided Navigation," Robotics and Automation, 10(1):11-22, February 1994. [41] J. Yu, L. McMillan, and S. Gortler, "Surface Camera Light Field Rendering," International Journal on Image and Graphics (IJIG), Special Issue on Integrated Image and Graphics Technologies, Volume 4, 2004. [42] J. Yu and L. McMillan, "General Linear Cameras," Proc. 8th European. Conf. on Computer Vision (ECCV 2004), Volume II 14-27. [43] J. Yu and L. McMillan, "A Framework for Multiperspective Rendering", Ren- dering Techniques 04, EurographicsSymposium on Rendering (EGSR 04):. 148 [44] J. Yu, J. Yang, and L. McMillan, "Real-time Reflection Mapping with Parallax", Proc. ACM SIGGRAPH Symposium on Interactive 3D Graphics (3D 2005), to appear. [45] J. Yu and L. McMillan, "Modelling Reflections via Multiperspective Imaging", Proc. IEEE Conf. Computer Vision and Pattern Recognition (2005), to appear. [46] J. Yu and L. McMillan, "Multiperspective Projection and Collineation ," Proc. ICCV 2005, to appear. [47] A. Zomet, D. Feldman, S. Peleg, and D. Weinshall, "Mosaicing New Views: The Crossed-Slits Projection," IEEE Trans. on PAMI (2003) 741-754. [48] J.Y. Zheng and S. Tsuji: Panoramic Representation for Route Recognition by a Mobile Robot. Int'l J. Computer Vision, vol. 9, no. 1 (1992) 55-76. 149 MITLibraries Document Services Room 14-0551 77 Massachusetts Avenue Cambridge, MA 02139 Ph: 617.253.5668 Fax: 617.253.1690 Email: docs@mit.edu http://libraries. mit.edu/docs DISCLAIMER OF QUALITY Due to the condition of the original material, there are unavoidable flaws in this reproduction. We have made every effort possible to provide you with the best copy available. If you are dissatisfied with this product and find it unusable, please contact Document Services as soon as possible. Thank you. Some pages in the original document contain color pictures or graphics that will not scan or reproduce well.

General Linear Cameras: Theory and Applications

Related documents

Products

Support

General Linear Cameras: Theory and Applications

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib