Lecture 3: Camera Rotations and Homographies X P Y Z Recap: Normalized Camera A camera is normalized if the units are chosen so that the focal length is 1. Then image coordinates (x,y) = X/Z,Y/Z In general, mapping from world points to image points uses the “calibration matrix”. This takes a 3D world point X,Y,Z and returns its 2d homogeneous pixel Computer Vision, Robert coordinates. Pless x p y (0,0,0) Camera Center of Projection x fx y 0 1 0 s fy 0 x0 X y 0 Y 1 Z Complete projection matrix: x fx y 0 1 0 . . . . KR . .Tx .T y . T z Computer Vision, Robert Pless s fy 0 x0 . y 0 . 1 . . R . X grid . T x Y grid .T y Z grid .Tz 1 x fx y 0 1 0 s fy 0 x0 . y 0 . 1 . . R . X grid . T x Y grid .T y Z grid .Tz 1 æ X grid ö æ x ö æ p1 p2 p3 p4 öç ÷ ÷çYgrid ÷ ç ÷ ç ç y ÷ @ ç p5 p6 p7 p8 ÷ç ÷ ÷÷ç Z grid ÷ ç1 ÷ çç p11 øç è ø è p9 p10 1 ÷ 1 è ø p1X +p2Y + p3Z + p4 x= p9X + P10Y + Z + p11 p9Xx + P10Yx + Zx + P11x = p1X + p2Y + p3Z + p4 p1X + p2Y + p3Z + p4 + 0 p5 + 0 p6 + 0 p7 + 0 p8 - Xxp9 -Yxp10 - xP11 = Zx p1X + p2Y + p3Z + p4 + 0 p5 + 0 p6 + 0 p7 + 0 p8 - Xxp9 -Yxp10 - xP11 = Zx 0 p1+ 0 p2 + 0 p3+ 0 p4 + Xp5 +Yp6 + Zp7 +1p8 - Xyp9 - Yyp10 - yP11 = Zy æ p1 ö ç ÷ ç p2 ÷ ç ÷ p3 ç ÷ ç p4 ÷ æ X Y Z 1 0 0 0 0 -Xx -Yx -x öç ÷ Zx p5 ç ÷ç ÷ ç 0 0 0 0 X Y Z 1 -Xy -Yy -y ÷ç p6 ÷ = Zy ç ÷ç ÷ ç ÷ç p7 ÷ è øç p8 ÷ ç ÷ ç p9 ÷ ç p10 ÷ ç ÷ ç p11 ÷ è ø x fx y 0 1 0 s fy 0 x0 . y 0 . 1 . . R . X grid . T x Y grid .T y Z grid .Tz 1 • Let’s consider a special case. Suppose we take two pictures of the world, from a camera at the same location. But the camera has rotated between the two pictures. x fx y 0 1 0 • First image, points in 3D, measured in x camera coordinate y 1 system. • Second image, camera has x' rotated. x 0 . y 0 . 1 . s fy 0 fx s 0 fy 0 0 fx y ' 0 1 0 s fy 0 x0 . y 0 . 1 . x 0 . y 0 . 1 . . R1 . . I . . R . X grid . T x Y grid .T y Z grid .Tz 1 X grid . 0 Y grid . 0 Z grid .0 1 X grid . 0 Y grid . 0 Z grid .0 1 • …mostly on chalkboard • On image 1 p = KP And image 2, p’ = KRP x fx p y 0 1 0 x fx p y 0 1 0 p = K((KR)-1)p’ = KR-1K-1p’ s fy 0 s fy 0 x0 . y 0 . 1 . x0 . y 0 . 1 . p KRP p KP p ' KRP . R1 . . R1 . X grid . 0 Y grid . 0 Z grid .0 1 . X grid . Y grid . Z grid p -1 -1 =KR K p’, so what? • Given a few examples of corresponding points p,p’, we can solve for a mapping KR-1K-1 of all points. Another Special Case:… Suppose the world is a plane. • Projection from the world to the image: x fx y 0 1 0 s fy 0 x0 y 0 1 R X grid T x Y grid T y Z grid Tz 1 Irrelevant 0 • Ignore z-coordinate (it is 0 anyway), drop the 3rd column of the 3x4 matrix, then you get a mapping between the plane and the image which is an arbitrary 3 x 3 matrix. Given 2 images: • p = KGP, • p’ = KG’P, so p’ = KG’(G-1K-1)p… p’ = (some 3x3) p Computer Vision, Robert Pless Homography: (x’,y’,1) ~ H (x,y,1) • Tells you exactly where the point goes. • Point (x,y) in one frame corresponds to point (x’,y’) in the other frame. If we need to think about multiple points, we may put subscripts on them. • Being careful about the homogenous coordinate, we write: x wx ' wy ' H y 1 w Computer Vision, Robert Pless Homography is a “simple” example of a 3D to 2D transformation It is also the “most complicated” linear 2D to 2D transformation. What other 2D 2D transformations are there? Computer Vision, Robert Pless Homography is most general, encompasses other transformations Projective 8 dof Affine 6 dof Similarity 4 dof h11 h 21 h 31 a 11 a 21 0 sr11 sr 21 0 h12 h 22 h 32 a 12 a 22 0 sr12 sr 22 r11 r12 Euclidean r r22 3 dof 21 Computer Vision, Robert 0 0 Pless 0 h13 h 23 h 33 tx ty 1 tx ty 1 tx ty 1 Views of a plane from different viewpoints, any view of a scene from the same viewpoint. Images of a “far away” object under any rotation Camera looking at an assembly line w/ zoom. Camera looking at an assembly line. Invariants… Projective 8 dof Affine 6 dof Similarity 4 dof Euclidean 3 dof h11 h 21 h 31 a 11 a 21 0 sr11 sr 21 0 r11 r 21 0 Computer Vision, Robert Pless h12 h 22 h 32 a 12 a 22 0 sr12 sr 22 0 r12 r22 0 h13 h 23 h 33 tx ty 1 tx ty 1 tx ty 1 Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints). Ratios of any edge lengths, angles. Absolute lengths, areas. Image registration Determining the 2d transformation that brings one into alignment (registers it) with another. Computer Vision, Robert Pless image Image Warping • What kind of warps are these? 0 . 6101 0 . 1680 0 . 0011 Computer Vision, Robert Pless 0 . 1918 0 . 6000 0 . 0010 34 . 2411 26 . 7034 1 How to solve for these mappings? Given: Solve for: 0 . 6101 0 . 1680 0 . 0011 Computer Vision, Robert Pless 0 . 1918 0 . 6000 0 . 0010 34 . 2411 26 . 7034 1 Unwrapping a matrix. x wx ' H y wy ' 1 w a d g c x wx ' wy ' f y w 1 1 b e h Write out the lines of this matrix equation. ax by c wx ' dx ey f wy ' gx hy 1 w And remember which variables are unknown. Computer Vision, Robert Pless Unwrapping a matrix. a d g b e h c x wx ' wy ' f y w 1 1 ax by c wx ' i dx ey f wy ' i gx hy 1 w ax by c 0 d 0 e 0 f 0 g 0 h wx i ' 0 a 0 b 0 c dx ey f 0 g 0 h wy i ' 0 a 0 b 0 c 0 d 0 e 0 f gx hy 1 w ax by c 0 d 0 e 0 f 0 g 0 h wx 0 ' 0 a 0 b 0 c dx ey f 0 g 0 h wy 0 ' Computer Vision, Robert Pless 0 a 0 b 0 c 0 d 0 e 0 f gx hy w 1 a d g b e h c x wx ' wy ' f y 1 1 w ax by c 0 d 0 e 0 f 0 g 0 h wx 0 ' 0 a 0 b 0 c dx ey f 0 g 0 h wy 0 ' 0 a 0 b 0 c 0 d 0 e 0 f gx hy w 1 Looks like another matrix equation: x 0 0 Computer Vision, Robert Pless y 1 0 0 0 0 0 0 0 x y 1 0 0 0 0 0 0 0 x y a b c ' xi d 0 ' yi e 0 1 f 1 g h w a d g b e h c x wx ' wy ' f y 1 1 w ax by c 0 d 0 e 0 f 0 g 0 h wx 0 ' 0 a 0 b 0 c dx ey f 0 g 0 h wy 0 ' 0 a 0 b 0 c 0 d 0 e 0 f gx hy w 1 Looks like another matrix equation: Data from different points Computer Vision, Robert Pless x 0 0 x 0 0 x 0 0 x 0 0 y 1 0 0 0 0 0 0 0 x y 1 0 0 0 0 0 0 0 x y y 1 0 0 0 0 0 0 0 x y 1 0 0 0 0 0 0 0 x y y 1 0 0 0 0 0 0 0 x y 1 0 0 0 0 0 0 0 x y y 1 0 0 0 0 0 0 0 x y 1 0 0 0 0 0 0 0 x y 0 a 0 ' y i 0 0 0 b 0 1 0 0 0 c 1 ' 0 xi 0 0 d 0 ' 0 yi 0 0 e 0 0 1 0 0 f 1 ' 0 0 0 xi 0 g ' 0 0 yi 0 h 0 0 0 1 0 w 1 ' 0 0 0 xi w 2 0 ' w 3 0 0 0 yi 0 0 0 0 1 w 4 1 xi ' 0 0 a d g b e h c x wx ' wy ' f y 1 1 w Challenges… • Maybe you have error in finding corresponding points, and want to use many many corresponding points. Then your number of unknowns keeps growing… • Is there a better way? • At the end of the day, how do we compute a real coordinate x’? Computer Vision, Robert Pless The game of “finding the linear constraint…” x’ = wx’ / w ax by c wx gx hy 1 w a d g ' x ' ax by c b e h c x wx ' f y wy ' 1 1 w Non-linear gx hy 1 x ( gx hy 1) ax by c ' Computer Vision, Robert Pless Linear (in a,b,c,g,h) x ( gx hy 1) ax by c ' gxx hyx x ax by c ' ' ' gxx hyx ax by c x ' ' ' ax by c gxx hyx x ' ' ' ax by c 0 d 0 e 0 f gxx hyx x ' 0 a 0 b 0 c dx ey f gxy hyy ' Computer Vision, Robert Pless ' ' y ' ' ax by c 0 d 0 e 0 f gxx i hyx i x i ' ' ' 0 a 0 b 0 c dx ey f gxy i hyy i y i ' x 0 ' y 1 0 0 0 xx ' yx ' 0 0 x y 1 xy ' yy ' And just add two more rows for each corresponding point Computer Vision, Robert Pless ' a b c d e f g h x' y ' x 0 y 1 0 0 0 xx ' yx ' 0 0 x y 1 xy ' yy ' a b c d e f g h x' y ' Ax=b Matlab: x = A\b Then make your homography matrix by rearranging x into a 3 x 3 matrix Computer Size ofVision, A? b?Robert Pless