Image Primitives and Correspondence
Stefano Soatto added with slides from Univ. of Maryland and
R.Bajcsy, UCB
Computer Science Department
University of California at Los Angeles
Image Primitives and Correspondence
Given an image point in left image, what is the (corresponding) point in the right
image, which is the projection of the same 3-D point
Siggraph 04
2
Image primitives and Features







The desirable properties of features are:
Invariance with respect to Grays scale/color
With respect to location (translation and
rotation)
With respect to scale
Robustness
Easy to compute
Local features vs. global features
Siggraph 04
3
Feature analysis
Points sensitive to illumination variation but fast to
compute
 Neighborhood features : gradient based (edge
detectors) measuring contrast ,robust to illumination
variation except for highlights
Fast computation ,it can be done in parallel.
The complimentary feature to gradient is region based.
The advantage of this feature is it can encompass
larger regions that are homogeneous and save
processing time.

Siggraph 04
4
Profiles of image intensity edges
Siggraph 04
5
Image gradient


The gradient of an image:
The gradient points in the direction of most rapid
change in intensity

The gradient direction is given by:

how does this relate to the direction of the edge?
The edge strength is given by the gradient magnitude

Siggraph 04
6
The discrete gradient
 How can we differentiate a digital image
f[x,y]?
 Option 1: reconstruct a continuous image,
then take gradient
 Option 2: take discrete derivative (finite
 How would you implement this as a crossdifference)
correlation?
Siggraph 04
7
Effects of noise


Consider a single row or column of the image
 Plotting intensity as a function of position
gives a signal
Where is the edge?
Siggraph 04
8
Solution: smooth first

Siggraph 04
Where is the edge?
 Look
for
9
2D edge detection filters
Laplacian of Gaussian
Gaussian

derivative of Gaussian
is the Laplacian operator:
Siggraph 04
10
Effect of  (Gaussian kernel size)
original

Canny with
Canny with
The choice of
depends on desired behavior
 large
detects large scale edges
04
 small
detects fineSiggraph
features
11
Scale
 Smoothing
 Eliminates noise edges.
 Makes edges smoother.
 Removes fine detail.
04
(Forsyth &Siggraph
Ponce)
12
Corner detection
Corners contain more edges than lines.

A point on a line is hard to match.
Siggraph 04
13
Corners contain more edges than lines.

A corner is easier
Siggraph 04
14
Edge Detectors Tend to Fail at Corners
Siggraph 04
15
Finding Corners
Intuition:
• Right at corner, gradient is ill defined.
• Near corner, gradient has two different
values.
Siggraph 04
16
Formula for Finding Corners
We look at matrix:
Gradient with respect to x,
times gradient with respect to y
Sum over a small region,
the hypothetical corner
 I
C
 I x I y
2
x
Matrix is symmetric
I I
I
x y
2
y



WHY THIS?
Siggraph 04
17
First, consider case where:
 I
C
 I x I y
2
x
I I
I
x y
2
y
 1 0 


  0 2 
What is region like if:
1. 1  0?
2. 2  0?
3. 1  0 and 2  0?
4. 1 > 0 and 2 > 0?
Siggraph 04
18
General Case:
From Linear Algebra, it follows that because C is
symmetric:

0


1
1
CR 
R

0

2

With R a rotation matrix.
So every case is like one on last slide.
Siggraph 04
19
So, to detect corners





Filter image.
Compute magnitude of the gradient
everywhere.
We construct C in a window.
Use Linear Algebra to find 1 and 2.
If they are both big, we have a corner.
Siggraph 04
20
Matching - Correspondence
Lambertian assumption
Rigid body motion
Correspondence
Siggraph 04
21
Local Deformation Models

Translational model

Affine model

Transformation of the intensity values and occlusions
Siggraph 04
22
Motion Field (MF)



The MF assigns a velocity vector to each pixel
in the image.
These velocities are INDUCED by the RELATIVE
MOTION btw the camera and the 3D scene
The MF can be thought as the projection of
the 3D velocities on the image plane.
Siggraph 04
23
Motion Field and Optical Flow Field

Motion field: projection of 3D motion vectors on image plane
Object point P0 has velocity v 0 , induces v i in image
dri
dt
r
r
r0 related to ri by i  0
f r0  zˆ 0
v0 


dr0
dt
v1 
Optical flow field: apparent motion of brightness patterns
We equate motion field with optical flow field
Siggraph 04
24
2 Cases Where this Assumption Clearly is not
Valid
(a) A smooth sphere is rotating
under constant illumination.
Thus the optical flow field
is zero, but the motion field
is not.
(a)
(b)
(b) A fixed sphere is
illuminated by a moving
source—the shading of the
image changes. Thus the
motion field is zero, but the
optical flow field is not.
Siggraph 04
25
Brightness Constancy Equation


Let P be a moving point in 3D:
 At time t, P has coords (X(t),Y(t),Z(t))
 Let p=(x(t),y(t)) be the coords. of its
image at time t.
 Let E(x(t),y(t),t) be the brightness at p at
time t.
Brightness Constancy Assumption:
 As P moves over time, E(x(t),y(t),t)
remains constant.
Siggraph 04
26
Brightness Constraint Equation
Let E  x, y, t  be the irradiance and u  x, y , v x, y  the components of optical flow.
E  x  ut , y  vt , t  t   E  x, y, t 
Taylor expansion
E
E
E
 y
 t
 e  E  x, y , t 
x
y
t
dividing by t and taking limit t  0
E  x , y , t   x
E dx E dy E


0
x dt y dt t
which is the expansion of the total derivative
dE
0
dt
short: E x u  E y v  Et  0
Siggraph 04
27
Brightness Constancy Equation
Taking derivative wrt time:
Siggraph 04
28
Brightness Constancy Equation
Let
(Frame spatial gradient)
(optical flow)
and
(derivative across frames)
Siggraph 04
29
Brightness Constancy Equation
Becomes:
vy
rE
-Et/|r E|
vx
The OF is CONSTRAINED to be on a line !
Siggraph 04
30
Interpretation
Values of (u, v) satisfying
the constraint equation lie
on a straight line in velocity
space. A local measurement
only provides this constraint
line (aperture problem).
Normal flow u n
E
x
, E y  u , v    Et
E E 
E , E 
T
Let n 
x
T
x
 E E
 E y Et 

u n  u  n n   2 x t , 2
E E E E 2
y
x
y 
 x
Siggraph 04
y
y
T
31
Aperture Problem
• Normal flow
Siggraph 04
32
Recall the corner detector
The matrix for corner detection:
is singular (not invertible) when det(ATA) = 0
But det(ATA) =  i = 0 -> one or both e.v. are 0
One e.v. = 0 -> no corner, just an edge
Two e.v. = 0 -> no corner, homogeneous region
Siggraph 04
Aperture
Problem !
33
Optical Flow
• Integrate over image patch
• Solve
Siggraph 04
34
Optical Flow, Feature Tracking
Conceptually:
rank(G) = 0 blank wall problem
rank(G) = 1 aperture problem
rank(G) = 2 enough texture – good feature candidates
In reality: choice of threshold is involved
Siggraph 04
35
Optical Flow
• Previous method - assumption locally constant flow
• Alternative regularization techniques (locally smooth flow fields,
integration along contours)
• Qualitative properties of the motion fields
Siggraph 04
36
Feature Tracking
Siggraph 04
37
3D Reconstruction - Preview
Siggraph 04
38
Harris Corner Detector - Example
Siggraph 04
39
Wide Baseline Matching
Siggraph 04
40
Region based Similarity Metric
• Sum of squared differences
• Normalize cross-correlation
• Sum of absolute differences
Siggraph 04
41
Edge Detection
original image
gradient magnitude
Canny edge detector
• Compute image derivatives
• if gradient magnitude >  and the value is a local maximum along gradient
direction – pixel is an edge candidate
Siggraph 04
42
Line fitting
y


x
Non-max suppressed gradient magnitude
• Edge detection, non-maximum suppression
(traditionally Hough Transform – issues of resolution, threshold
selection and search for peaks in Hough space)
• Connected components on edge pixels with similar orientation
- group pixels with common orientation
Siggraph 04
43
Line Fitting
second moment matrix
associated with each
connected component
• Line fitting Lines determined from eigenvalues and eigenvectors of A
• Candidate line segments - associated line quality
Siggraph 04
44
Take home messages




Correspondence is easy/difficult/impossible
depending on the imaging constraints
Correspondence and reconstruction are
tightly coupled problems, can be solved
simultaneously [Jin et al., CVPR 2004]
For most scenes simple descriptors suffice to
establish a few (50-500) corresponding
points/lines
From now on just geometry
Siggraph 04
45