CS 485 / 685
Computer Vision
Instructor: Mircea Nicolescu
Lecture 10
Why Are Interest Points Useful?
• For establishing corresponding points between
images
panorama stitching
stereo matching
left camera
right camera
2
How to Find Corresponding Points?
?
()
feature
descriptor
?
=
()
feature
descriptor
• Need to define local patches surrounding the interest
points and extract feature descriptors from every patch.
• Match feature descriptors to find corresponding points.
3
Properties of Good Features
• Local: features are local, so robust to
occlusion and clutter (no prior segmentation!)
• Accurate: precise localization
• Invariant (or covariant)
• Robust: noise, blur, compression, etc.
do not have a big impact on the feature
Repeatable
• Distinctive: individual features can be
matched to a large database of objects
• Efficient: close to real-time performance
4
Invariance / Covariance
• A function f is invariant under a transformation T if
its value does not change when the transformation
is applied to its argument:
if f(x) = y then f(T(x))=y
• A function f is covariant when it changes in a way
consistent with the transformation T:
if f(x) = y then f(T(x))=T(f(x))=T(y)
5
Interest Point Detectors Should Be Covariant
• Features should be detected in corresponding locations
despite geometric or photometric changes.
6
Interest Point Descriptors Should Be Invariant
?
()
feature
descriptor
?
=
()
feature
descriptor
Should be similar despite geometric
or photometric transformations
7
Interest Point Candidates
Use features with gradients in at least two, significantly
different orientations (e.g., corners, junctions etc.)
8
Aperture Problem
A point on a line is hard to match
t
t+1
A corner is easier to match
t
t+1
9
Interest Point Candidates
auto-correlation
10
Steps in Corner Detection
1. For each pixel, the corner operator is applied to obtain a
cornerness measure for this pixel.
2. Threshold cornerness map to eliminate weak corners.
3. Apply non-maximal suppression to eliminate points whose
cornerness measure is not larger than the cornerness values of all
points within a certain distance.
11
Steps in Corner Detection
12
Corner Detection Methods
• Contour based
− Extract contours and search for maximal curvature or
inflexion points (i.e., curvature zero-crossing) along the
contour.
• Intensity based
− Compute a measure that indicates the presence of an
interest point
− (1) directly from gray (or color) values, or
− (2) by first fitting a parametric model to the gray (or color)
values.
− Methods using parametric models can localize corners
to sub-pixel accuracy but are more expensive.
13
Curvature Scale Space
A contour-based approach: curvature scale space
• Parametric contour representation:
(x(t), y(t))
Curvature:
14
Curvature Scale Space
g(t,σ):
Gaussian
Curvature:
15
Curvature Scale Space
σ
G. Bebis, G. Papadourakis and S. Orphanoudakis, "Curvature Scale Space Driven
Object Recognition with an Indexing Scheme based on Artificial Neural Networks",
Pattern Recognition, Vol. 32, No. 7, pp. 1175-1201, 1999.
16
Zuniga-Haralick Detector
• Approximate image function in the neighborhood
of the pixel (i,j) by a bi-cubic polynomial.
measure of "cornerness”:
 2(k 22 k6  k 2 k3k5  k32 k 4 )
ZH (i, j ) 
(k 22  k32 )3 / 2
17
Corner Detection Using Edge Detection?
• Edge detectors are not stable at corners.
• Gradient is ambiguous at corner tip (i.e.,
discontinuous near corners).
18
Corner Detection Using Intensity
• Image gradient has two or more dominant directions near a
corner.
• Shifting a window in any direction should give a large
change in intensity.
“flat” region:
no change in all
directions
“edge”: no change
along the edge
direction
“corner”: significant
change in all
directions
19
Moravec Detector
• Idea: measure intensity variation at (x,y) by
shifting a small window (3x3 or 5x5) by one pixel
in each of the eight principal directions
(horizontally, vertically, and four diagonals).
20
Moravec Detector
• The intensity variation Sw in a given direction (∆x, ∆y)
can be calculated as follows:
21
Moravec Detector
• Moravec’s detector calculates Sw in all 8
directions:
∆x, ∆y in {-1,0,1}
SW(-1,-1), SW(-1,0), ..., SW(1,1)
22
Moravec Detector
• The “cornerness” of a pixel is the minimum intensity
variation found over the eight shift directions:
Cornerness(x,y) = min{SW(-1,-1), SW(-1,0), ...SW(1,1)}
Cornerness
Map
(normalized)
Note response to isolated points!
23
Moravec Detector
• Non-maximal suppression will yield final corners
24
Moravec Detector
• Does a reasonable job in
finding the majority of true
corners.
• Edge points not in one of the
eight principal directions
will be assigned a relatively
large cornerness value.
25
Moravec Detector
• The response is anisotropic as the intensity
variation is only calculated at a discrete set of
directions and shifts (i.e., not rotationally
invariant)
26
Harris Detector
•
Improves the Moravec detector by:
(1) Avoiding the use of discrete directions and discrete
shifts
(2) Using a Gaussian window instead of a square window
1 in window, 0 outside
Gaussian
27
Harris Detector
• Idea: decompose SW by factoring out ∆x, ∆y
• Use Taylor series expansion to “linearize”
28
Taylor Series Expansion –Review
• A representation of a function f(x) as an infinite
sum of terms calculated from the values of its
derivatives at a single point a:
1D:
f´(a)
f´´(a)
f ( x )  f(a) 
( x  a) 
( x  a)2 
1!
2!
f ( n ) (a )

( x  a)n  O( x n 1 )
n!
29
Taylor Series Expansion – Example
The exponential function (in blue),
and the sum of the first n+1 terms
of its Taylor series at a=0 (in red).
30
Taylor Series Expansion –Review
• Can be generalized to higher dimensions:
2D:
1
f ( x)  f (a)  f (a)( x  a)  ( x  a)T Hf (a)( x  a )  ...
2
Gradient
Hessian
Hessian in
n dimensions
31
Harris Detector
Using first order
Taylor approximation:
f ( x)  f (a)  f (a)( x  a)
substitute
32
Harris Detector
SW
rewrite using dot product
33
Harris Detector
Since
:
34
Harris Detector
 x 
So: SW (x, y )   x y  AW ( x, y )  
 y 
(
where
AW(x,y)=
f ( xi , yi ) 2
)
x
(
or
 f x2
AW ( x, y )   
xW , yW 
 fx f y
f ( xi , yi ) 2
)
y
fx f y 
2 
f y 
AW(x,y) is a 2 x 2 matrix called auto-correlation or 2nd order
moment matrix
35
Harris Detector
• Using a window function w(x,y), we rewrite SW :
w(x,y) :
1 in window, 0 outside
SW (x, y )   w( xi , yi )  f ( xi , yi )  f ( xi  x, yi  y ) 
2
xi , yi
36
Harris Detector
•Then Aw becomes:
 f x2
AW ( x, y )   
xW , yW 
 fx f y
 f x2
AW   w( x, y ) 
x, y
 f x f y

2
w
(
x
,
y
)
f

x
f x f y   x , y

2 
f y  
 w( x, y ) f x f y
 x, y
fx f y 
2 
f y 

w
(
x
,
y
)
f
f

x y
x, y


2
w( x, y ) f y 

x, y

37
Harris Detector
• Harris uses a Gaussian window: w(x,y)=G(x,y,σI)
where σI is called the “integration” scale:
w(x,y) :
Gaussian
 f x2
AW   w( x, y ) 
x, y
 f x f y

2
w
(
x
,
y
)
f

x
f x f y   x , y

2 
f y  
 w( x, y ) f x f y
 x, y

w
(
x
,
y
)
f
f

x y
x, y


2
w( x, y ) f y 

x, y

38
Properties of Auto-Correlation Matrix
Describes the gradient distribution
(i.e., local structure) inside the
window!
 f x2
AW   w( x, y ) 
x, y
 f x f y
Does not depend on
fx f y 

f y2 
x, y
39
Properties of Auto-Correlation Matrix
Aw is symmetric and
can be decomposed:
1 0 
AW  R 
R

 0 2 
1
• We can visualize AW as an ellipse with axis lengths and
directions determined by its eigenvalues and eigenvectors.
(min)1/2
(max)1/2
40
Harris Detector
• The eigenvectors of AW encode
direction of intensity change.
• The eigenvalues of AW encode
strength of intensity change.
v1
v2
direction of the
slowest change
(min)1/2
(max)1/2
direction of the
fastest change
41
Harris Detector
2
“Edge”
2 >> 1
Classification of
pixels using the
eigenvalues of AW :
1 and 2 are small;
SW is almost
constant in all
directions
“Flat”
region
“Corner”
1 and 2 are large,
1 ~ 2;
intensity changes in
all directions
“Edge”
1 >> 2
1
42
Distribution of fx and fy
fy
fx
fy
fy
fx
fx
43
Harris Detector
One way to compute “good” corners is by
thresholding min(λ1,λ2)
%
assume
1 > 2
2
J. Shi and C. Tomasi, "Good Features to Track”, 9th IEEE Conference on
Computer Vision and Pattern Recognition, June 1994.
44