advertisement

Image Primitives and Correspondence Stefano Soatto added with slides from Univ. of Maryland and R.Bajcsy, UCB Computer Science Department University of California at Los Angeles Image Primitives and Correspondence Given an image point in left image, what is the (corresponding) point in the right image, which is the projection of the same 3-D point Siggraph 04 2 Image primitives and Features The desirable properties of features are: Invariance with respect to Grays scale/color With respect to location (translation and rotation) With respect to scale Robustness Easy to compute Local features vs. global features Siggraph 04 3 Feature analysis Points sensitive to illumination variation but fast to compute Neighborhood features : gradient based (edge detectors) measuring contrast ,robust to illumination variation except for highlights Fast computation ,it can be done in parallel. The complimentary feature to gradient is region based. The advantage of this feature is it can encompass larger regions that are homogeneous and save processing time. Siggraph 04 4 Profiles of image intensity edges Siggraph 04 5 Image gradient The gradient of an image: The gradient points in the direction of most rapid change in intensity The gradient direction is given by: how does this relate to the direction of the edge? The edge strength is given by the gradient magnitude Siggraph 04 6 The discrete gradient How can we differentiate a digital image f[x,y]? Option 1: reconstruct a continuous image, then take gradient Option 2: take discrete derivative (finite How would you implement this as a crossdifference) correlation? Siggraph 04 7 Effects of noise Consider a single row or column of the image Plotting intensity as a function of position gives a signal Where is the edge? Siggraph 04 8 Solution: smooth first Siggraph 04 Where is the edge? Look for 9 2D edge detection filters Laplacian of Gaussian Gaussian derivative of Gaussian is the Laplacian operator: Siggraph 04 10 Effect of (Gaussian kernel size) original Canny with Canny with The choice of depends on desired behavior large detects large scale edges 04 small detects fineSiggraph features 11 Scale Smoothing Eliminates noise edges. Makes edges smoother. Removes fine detail. 04 (Forsyth &Siggraph Ponce) 12 Corner detection Corners contain more edges than lines. A point on a line is hard to match. Siggraph 04 13 Corners contain more edges than lines. A corner is easier Siggraph 04 14 Edge Detectors Tend to Fail at Corners Siggraph 04 15 Finding Corners Intuition: • Right at corner, gradient is ill defined. • Near corner, gradient has two different values. Siggraph 04 16 Formula for Finding Corners We look at matrix: Gradient with respect to x, times gradient with respect to y Sum over a small region, the hypothetical corner I C I x I y 2 x Matrix is symmetric I I I x y 2 y WHY THIS? Siggraph 04 17 First, consider case where: I C I x I y 2 x I I I x y 2 y 1 0 0 2 What is region like if: 1. 1 0? 2. 2 0? 3. 1 0 and 2 0? 4. 1 > 0 and 2 > 0? Siggraph 04 18 General Case: From Linear Algebra, it follows that because C is symmetric: 0 1 1 CR R 0 2 With R a rotation matrix. So every case is like one on last slide. Siggraph 04 19 So, to detect corners Filter image. Compute magnitude of the gradient everywhere. We construct C in a window. Use Linear Algebra to find 1 and 2. If they are both big, we have a corner. Siggraph 04 20 Matching - Correspondence Lambertian assumption Rigid body motion Correspondence Siggraph 04 21 Local Deformation Models Translational model Affine model Transformation of the intensity values and occlusions Siggraph 04 22 Motion Field (MF) The MF assigns a velocity vector to each pixel in the image. These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene The MF can be thought as the projection of the 3D velocities on the image plane. Siggraph 04 23 Motion Field and Optical Flow Field Motion field: projection of 3D motion vectors on image plane Object point P0 has velocity v 0 , induces v i in image dri dt r r r0 related to ri by i 0 f r0 zˆ 0 v0 dr0 dt v1 Optical flow field: apparent motion of brightness patterns We equate motion field with optical flow field Siggraph 04 24 2 Cases Where this Assumption Clearly is not Valid (a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not. (a) (b) (b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not. Siggraph 04 25 Brightness Constancy Equation Let P be a moving point in 3D: At time t, P has coords (X(t),Y(t),Z(t)) Let p=(x(t),y(t)) be the coords. of its image at time t. Let E(x(t),y(t),t) be the brightness at p at time t. Brightness Constancy Assumption: As P moves over time, E(x(t),y(t),t) remains constant. Siggraph 04 26 Brightness Constraint Equation Let E x, y, t be the irradiance and u x, y , v x, y the components of optical flow. E x ut , y vt , t t E x, y, t Taylor expansion E E E y t e E x, y , t x y t dividing by t and taking limit t 0 E x , y , t x E dx E dy E 0 x dt y dt t which is the expansion of the total derivative dE 0 dt short: E x u E y v Et 0 Siggraph 04 27 Brightness Constancy Equation Taking derivative wrt time: Siggraph 04 28 Brightness Constancy Equation Let (Frame spatial gradient) (optical flow) and (derivative across frames) Siggraph 04 29 Brightness Constancy Equation Becomes: vy rE -Et/|r E| vx The OF is CONSTRAINED to be on a line ! Siggraph 04 30 Interpretation Values of (u, v) satisfying the constraint equation lie on a straight line in velocity space. A local measurement only provides this constraint line (aperture problem). Normal flow u n E x , E y u , v Et E E E , E T Let n x T x E E E y Et u n u n n 2 x t , 2 E E E E 2 y x y x Siggraph 04 y y T 31 Aperture Problem • Normal flow Siggraph 04 32 Recall the corner detector The matrix for corner detection: is singular (not invertible) when det(ATA) = 0 But det(ATA) = i = 0 -> one or both e.v. are 0 One e.v. = 0 -> no corner, just an edge Two e.v. = 0 -> no corner, homogeneous region Siggraph 04 Aperture Problem ! 33 Optical Flow • Integrate over image patch • Solve Siggraph 04 34 Optical Flow, Feature Tracking Conceptually: rank(G) = 0 blank wall problem rank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates In reality: choice of threshold is involved Siggraph 04 35 Optical Flow • Previous method - assumption locally constant flow • Alternative regularization techniques (locally smooth flow fields, integration along contours) • Qualitative properties of the motion fields Siggraph 04 36 Feature Tracking Siggraph 04 37 3D Reconstruction - Preview Siggraph 04 38 Harris Corner Detector - Example Siggraph 04 39 Wide Baseline Matching Siggraph 04 40 Region based Similarity Metric • Sum of squared differences • Normalize cross-correlation • Sum of absolute differences Siggraph 04 41 Edge Detection original image gradient magnitude Canny edge detector • Compute image derivatives • if gradient magnitude > and the value is a local maximum along gradient direction – pixel is an edge candidate Siggraph 04 42 Line fitting y x Non-max suppressed gradient magnitude • Edge detection, non-maximum suppression (traditionally Hough Transform – issues of resolution, threshold selection and search for peaks in Hough space) • Connected components on edge pixels with similar orientation - group pixels with common orientation Siggraph 04 43 Line Fitting second moment matrix associated with each connected component • Line fitting Lines determined from eigenvalues and eigenvectors of A • Candidate line segments - associated line quality Siggraph 04 44 Take home messages Correspondence is easy/difficult/impossible depending on the imaging constraints Correspondence and reconstruction are tightly coupled problems, can be solved simultaneously [Jin et al., CVPR 2004] For most scenes simple descriptors suffice to establish a few (50-500) corresponding points/lines From now on just geometry Siggraph 04 45