Pinhole camera model ((X,Y,Z) , , ) P origin p (x,y) (optical center) • principal point The coordinate system – We W will ill use the th pin-hole i h l model d l as an approximation i ti – Put the optical center (Center Of Projection) at the origin – Put the image plane (Projection Plane) in front of the COP (Why?) Pinhole camera model fX x= Z fY y= Z principal p p point ⎛ x ⎞ ⎛ fX ⎞ ⎡ f ⎜ ⎟ ⎜ ⎟ ⎢ ⎜ y ⎟ ~ ⎜ fY ⎟ = ⎢ 0 ⎜1⎟ ⎜ Z ⎟ ⎢0 ⎝ ⎠ ⎝ ⎠ ⎣ 0 f 0 0⎤ ⎛ X ⎞ ⎜ ⎟ ⎥ 0⎥ ⎜ Y ⎟ 1⎥⎦⎝⎜ Z ⎟⎠ Pinhole camera model ((X,Y,Z) , , ) P origin p (x,y) (optical center) principal point y x ⎛ x ⎞ ⎛ fX ⎞ ⎡ f ⎜ ⎟ ⎜ ⎟ ⎢ ⎜ y ⎟ ~ ⎜ fY ⎟ = ⎢ 0 ⎜1⎟ ⎜ Z ⎟ ⎢0 ⎝ ⎠ ⎝ ⎠ ⎣ 0 f 0 0⎤ ⎛ X ⎞ ⎜ ⎟ ⎥ 0⎥ ⎜ Y ⎟ 1⎦⎥⎜⎝ Z ⎟⎠ Pinhole camera model ((X,Y,Z) , , ) P origin p (x,y) (optical center) principal point y x ⎛ x ⎞ ⎛ fX ⎞ ⎡ f ⎜ ⎟ ⎜ ⎟ ⎢ ⎜ y ⎟ ~ ⎜ fY ⎟ = ⎢ 0 ⎜1⎟ ⎜ Z ⎟ ⎢0 ⎝ ⎠ ⎝ ⎠ ⎣ 0 f 0 x0 ⎤⎛ X ⎞ ⎜ ⎟ ⎥ y0 ⎥ ⎜ Y ⎟ 1 ⎦⎥⎜⎝ Z ⎟⎠ Intrinsic matrix Is this form of K good enough? ⎡f K = ⎢⎢ 0 ⎢⎣ 0 0 f 0 xc ⎤ yc ⎥⎥ 1 ⎥⎦ • non-square pixels (digital video) ⎡ fx K = ⎢⎢ 0 ⎢⎣ 0 0 fy 0 xc ⎤ yc ⎥⎥ 1 ⎥⎦ Intrinsic matrix Is this form of K good enough? ⎡f K = ⎢⎢ 0 ⎢⎣ 0 • non-square pixels (digital video) • skew ⎡f x K = ⎢⎢ 0 ⎢⎣ 0 0 f 0 s fy 0 xc ⎤ yc ⎥⎥ 1 ⎥⎦ xc ⎤ yc ⎥⎥ 1 ⎥⎦ Intrinsic matrix ⎡f K = ⎢⎢ 0 ⎢⎣ 0 0 • non-square pixels (digital video) • skew ⎡ fx • radial distortion K=⎢0 s fy Is this form of K good enough? ⎢ ⎢⎣ 0 0 f 0 xc ⎤ yc ⎥⎥ 1 ⎥⎦ xc ⎤ yc ⎥⎥ 1 ⎥⎦ Distortion • Radial distortion of the image g – Caused by imperfect lenses – Deviations are most noticeable for rays that pass through the edge of the lens Barrel Distortion No distortion Wide Angle Lens Barrel Pin Cushion Distortion No distortion Telephoto lens Pin cushion Modeling distortion Distortion-Free: fX x= Z fY y= Z With Distortion: 1. Project (X, Y, Z) to “normalized” normalized image coordinates X xn = Z Y yn = Z r 2 = xn2 + yn2 2. Apply radial distortion ( ) y = y (1 + κ r + κ r ) xd = xn 1 + κ 1r 2 + κ 2 r 4 2 d 3. Apply pp y focal length g translate image center n x = fxd + xc y = fyd + yc • To model lens distortion – Use above projection operation instead of standard projection matrix multiplication 1 4 2 Camera rotation and translation ⎛ X '⎞ ⎛X⎞ ⎜ ⎟ ⎜ ⎟ ⎜ Y ' ⎟ = R 3×3 ⎜ Y ⎟ + t ⎜ Z'⎟ ⎜Z⎟ ⎝ ⎠ ⎝ ⎠ ⎛ x⎞ ⎡ f ⎜ ⎟ ⎢ ⎜ y⎟ ~ ⎢0 ⎜1⎟ ⎢0 ⎝ ⎠ ⎣ 0 f 0 ⎛X⎞ ⎜ ⎟ xc ⎤ ⎜Y ⎟ ⎥ yc ⎥[R t ]⎜ ⎟ Z ⎜ ⎟ 1 ⎥⎦ ⎜1⎟ ⎝ ⎠ x ~ K [R t ]X extrinsic matrix Two kinds of parameters • internal or intrinsic parameters: focal length, optical center, skew • external or extrinsic (p (pose): ) rotation and translation: ⎛ x⎞ ⎡ f ⎜ ⎟ ⎢ ⎜ y⎟ ~ ⎢0 ⎜1⎟ ⎢0 ⎝ ⎠ ⎣ 0 f 0 ⎛X⎞ ⎜ ⎟ ⎡a xc ⎤ ⎜Y ⎟ ⎢ ⎥ yc ⎥[R t ]⎜ ⎟ = ⎢ e Z ⎜ ⎟ ⎢⎣ i 1 ⎥⎦ ⎜1⎟ ⎝ ⎠ b f c g j k ⎛X⎞ d ⎤⎜ ⎟ ⎜Y ⎟ ⎥ h ⎥⎜ ⎟ Z l ⎥⎦⎜⎜ ⎟⎟ ⎝1⎠ Other projection models Orthographic projection • Special case of perspective projection – Distance from the COP to the PP is infinite Image World – Also called “parallel projection”: (x, y, z) → (x, y) Other types of projections • Scaled orthographic – Also called “weak perspective” • Affine projection – Also called “paraperspective” Fun with perspective Perspective cues Perspective cues Fun with perspective Ames room Forced perspective in LOTR Elijah Wood: 5' 6" (1.68 m) Ian McKellen 5' 11" (1.80 m) Camera calibration Camera calibration ⎛ x⎞ ⎡ f ⎜ ⎟ ⎢ ⎜ y⎟ ~ ⎢0 ⎜1⎟ ⎢0 ⎝ ⎠ ⎣ 0 f 0 ⎛X⎞ ⎜ ⎟ ⎡a xc ⎤ ⎜Y ⎟ ⎢ ⎥ yc ⎥[R t ]⎜ ⎟ = ⎢ e Z ⎜ ⎟ ⎢⎣ i 1 ⎥⎦ ⎜1⎟ ⎝ ⎠ b c f j g k ⎛X⎞ d ⎤⎜ ⎟ ⎜Y ⎟ ⎥ h ⎥⎜ ⎟ Z l ⎥⎦⎜⎜ ⎟⎟ ⎝1⎠ • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Using objects with known geometry as reference 2. Self calibration (structure from motion) Camera calibration approaches • Directly estimate 11 unknowns in the M matrix using known 3D points ((Xi,Yi,Zi) , , ) and measured feature positions (ui,vi) x ~ K [R t ]X = MX Linear regression Linear regression ⎡Xi ⎢0 ⎣ Yi Zi 1 0 0 0 0 − ui X i − uiYi 0 0 0 Xi Yi Zi 1 − vi X i − viYi ⎡m00 ⎤ ⎢m ⎥ ⎢ 01 ⎥ ⎢m02 ⎥ ⎢ ⎥ ⎢ m03 ⎥ ⎢ m10 ⎥ − ui Z i ⎤ ⎢ ⎥ ⎡ui ⎤ ⎢ m11 ⎥ = ⎢ ⎥ ⎥ − viYi ⎦ ⎢ ⎥ ⎣ vi ⎦ m ⎢ 12 ⎥ ⎢ m13 ⎥ ⎢m ⎥ ⎢ 20 ⎥ ⎢ m21 ⎥ ⎢ ⎥ ⎣m22 ⎦ Linear regression X 1 Y1 Z1 1 0 0 0 ⎡ ⎢ 0 0 0 0 X 1 Y1 Z1 ⎢ M ⎢ 0 0 ⎢ X N YN Z N 1 0 ⎢ 0 0 0 0 X N YN Z N ⎣ 0 1 0 1 ⎡ m00 ⎤ ⎢m ⎥ ⎢ 01 ⎥ ⎢ m02 ⎥ ⎢ ⎥ − u1 X 1 − u1Y1 − u1Z1 ⎤ ⎢ m03 ⎥ ⎥ ⎢ m10 ⎥ − v1 X 1 − v1Y1 − v1Y1 ⎥ ⎢ ⎥ ⎡ui ⎤ ⎥ ⎢ m11 ⎥ = ⎢ v ⎥ − u N X N − u N YN − u N Z N ⎥ ⎢ ⎥ ⎣ i ⎦ m12 ⎢ ⎥ ⎥ − v N X N − v N YN − v N YN ⎦ ⎢ m13 ⎥ ⎢m ⎥ ⎢ 20 ⎥ ⎢ m21 ⎥ ⎢ ⎥ ⎣m22 ⎦ Solve for Projection Matrix M using least leastsquare techniques Normal equation (Geometric Interpretation) Given an overdetermined system Ax = b the normal equation is that which minimizes the sum of the square differences between left and right sides min Ax − b 2 ⇔ A ( Ax − b) = 0 −1 T T ⇔x= A A A b T ( ) Normal equation (Differential Interpretation) E (x) = Ax − b 2 ⎡ a11 ... a1m ⎤ ⎡ b1 ⎤ ⎢ : ⎥ ⎢ ⎥ x : : ⎡ ⎤ 1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ : : ⎥⎢ : ⎥ = ⎢ : ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ x : : : m ⎣ ⎦ ⎢ ⎥ ⎢ ⎥ ⎢⎣an1 ... anm ⎥⎦ ⎢⎣bn ⎥⎦ nxm, n equations, i m variables i bl Normal equation E (x) = Ax − b 2 = (Ax − b ) (Ax − b ) T = (x A − b )(Ax − b ) T T T = x A Ax − b Ax − x A b + b b T T T ( T ) T T ( T ) T Carl Friedrich Gauss = x A Ax − A b x − A b x + b b T T T ∂E T T = 2 A Ax − 2 A b ∂x T T Who h invented i d Least Square? S ? Nonlinear optimization • A probabilistic view of least square • Feature measurement equations • Likelihood of M given {(ui,vi)} Optimal estimation • Log likelihood of M given {(ui,vi)} 2 m00 X i + m01Yi + m02 Z i + m03 ⎞ m10 X i + m11Yi + m12 Z i + m13 ⎞ 1 ⎛ 1 ⎛ ⎟⎟ + 2 ⎜⎜ vi − ⎟⎟ = ∑ 2 ⎜⎜ ui − m20 X i + m21Yi + m22 Z i + 1 ⎠ m20 X i + m21Yi + m22 Z i + 1 ⎠ σ i ⎝ i =1 σ i ⎝ N • It is a least square problem (but not necessarily linear least square) • How do we minimize C? 2 Nonlinear least square methods Least square fitting number of data points number of parameters Nonlinear least square fitting Function minimization Least square is related to function minimization. It is very hard to solve in general. Here, we only consider a simpler p p problem of finding g local minimum. Function minimization Quadratic functions Approximate the function with a quadratic function within a small neighborhood Function minimization Recap: Calibration • Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui (ui,vi) vi) x ~ K [R t ]X = MX Camera Model: Recap: Calibration • Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui (ui,vi) vi) x ~ K [R t ]X = MX Linear Approach: Recap: Calibration • Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui (ui,vi) vi) x ~ K [R t ]X = MX NonLinear Approach: 2 ⎛ m00 X i + m01Yi + m02 Z i + m03 ⎞ ⎛ m10 X i + m11Yi + m12 Z i + m13 ⎞ ⎜⎜ ui − ⎟⎟ + ⎜⎜ vi − ⎟⎟ ∑ m20 X i + m21Yi + m22 Z i + 1 ⎠ ⎝ m20 X i + m21Yi + m22 Z i + 1 ⎠ i =1 ⎝ N 2 Practical Issue is hard to make and the 3D feature positions are difficult to measure! A popular calibration tool Multi-plane calibration Advantage Images courtesy Jean-Yves Bouguet, Intel Corp. • Only requires a plane • Don Don’tt have to know positions/orientations • Good code available online! – Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/ – Matlab version by Jean-Yves Jean Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html – Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/ Step 1: data acquisition Step 2: specify corner order Step 3: corner extraction Step 3: corner extraction Step 4: minimize projection error Step 4: camera calibration Step 4: camera calibration Step 5: refinement