CS 485 / 685
Computer Vision
Instructor: Mircea Nicolescu
Lecture 11
Harris Detector
2
“Edge”
2 >> 1
Classification of
pixels using the
eigenvalues of AW :
1 and 2 are small;
SW is almost
constant in all
directions
“Flat”
region
“Corner”
1 and 2 are large,
1 ~ 2;
intensity changes in
all directions
“Edge”
1 >> 2
1
2
Distribution of fx and fy
fy
fx
fy
fy
fx
fx
3
Harris Detector
One way to compute “good” corners is by
thresholding min(λ1,λ2)
%
assume
1 > 2
2
J. Shi and C. Tomasi, "Good Features to Track”, 9th IEEE Conference on
Computer Vision and Pattern Recognition, June 1994.
4
Harris Detector
• To avoid computing the eigenvalues explicitly, the
Harris detector uses the following function:
R(AW) = det(AW) – α trace2(AW)
α is a const
which is equal to:
R(AW) = λ1 λ2- α (λ1+ λ2)2
5
Harris Detector
Classification of image
points using R(AW):
“Edge”
R<0
“Corner”
R>0
R(AW) = det(AW) – α trace2(AW)
α is usually between
0.04 and 0.06
|R| small
“Flat”
region
“Edge”
R<0
6
Harris Detector
• Other functions:
2  a1
12
det A

trA
1  2
7
Harris Detector – Steps
 f x2
AW   w( x, y ) 
x, y
 f x f y
f ( xi , yi ) 
 G ( x, y,  D ) * f ( xi , yi )
x
x
f ( xi , yi ) 
 G( x, y,  D )* f ( xi , yi )
y
y
fx f y 

f y2 
σD is called the
“differentiation” scale
8
Harris Detector – Steps
 f x2
AW   w( x, y ) 
x, y
 f x f y
fx f y 

f y2 
(using σI)
R(AW) = det(AW) – α trace2(AW)
9
Harris Detector – Example
10
Harris Detector – Scale Parameters
• Harris detector requires two scale parameters:
(i) a differentiation scale σD for smoothing prior to the
computation of image derivatives,
and
(ii) an integration scale σI for defining the size of the
Gaussian window (i.e., integrating derivative
responses).
AW(x,y)  AW(x,y,σI,σD)
• Typically, σI=γσD
11
Harris Detector – Example
12
Compute Corner Response R
13
Find points with large corner response: R>threshold
14
Take only points of local maxima of R
15
Map corners on original image (for visualization)
16
Invariance to Geometric/Photometric Changes
• Is the Harris detector invariant to geometric and
photometric changes?
− Rotation
− Scale
− Affine
− Linear intensity change: I(x,y)  a I(x,y) + b
17
Harris Detector: Rotation Invariance
• Rotation
Ellipse rotates but its shape (i.e. eigenvalues)
remains the same
Corner response R is invariant to image rotation
18
Harris Detector: Rotation Invariance
19
Harris Detector: Photometric Changes
• Linear intensity change:
- Only derivatives are used => invariance to intensity
shift I(x,y)  I (x,y) + b
- Intensity scale: I(x,y)  a I(x,y)
R
R
threshold
x (image coordinate)
x (image coordinate)
Partially invariant to linear intensity change
20
Harris Detector: Scale Invariance
• Scaling
Corner
All points will be
classified as edges
rather than corners.
Not invariant to scaling (and affine transforms)
21
Harris Detector: Repeatability
22
Harris Detector: Repeatability
23
Harris Detector: Repeatability
When do two points correspond?
24
Harris Detector: Repeatability
How do we find correspondences?
25
Harris Detector: Repeatability
• In a comparative study of different interest point detectors,
Harris was shown to be the most repeatable.
(σI=1, σD=2)
C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors",
International Journal of Computer Vision 37(2), 151.172, 2000.
26
Harris Detector: Disadvantages
• Sensitive to:
− Scale change
− Significant viewpoint change
− Significant contrast change
27
How to Handle Scale Changes?
•
AW(x,y,σI,σD) must be adapted to scale changes.
•
If scale change is known, adapt the Harris
detector to it by properly setting σI and σD.
•
If scale change is unknown, detect interest points
at multiple scales.
28
Multi-scale Harris Detector
• Detects interest points at varying scales.
R(AW) = det(AW(x,y,σI,σD)) – α trace2(AW(x,y,σI,σD))
scale
σn=knσ
σD= σn
σI=γσD
σn
y
 Harris 
x
29
Multi-scale Harris Detector
Interest points detected at varying scales:
M. Brown, R. Szeliski, and S. Winder, “Multi-image matching using multi-scale
oriented patches”, IEEE Conference on Computer Vision and Pattern Recognition,
vol. I, pages 510-517, 2005.
30
Multi-scale Harris Detector
•
•
The same interest point will be detected at
multiple consecutive scales.
Interest point location will shift as scale increases
(due to smoothing).
The size of each circle
corresponds to the scale at
which the interest point
was detected.
31
How to Match Them?
•
•
Corresponding features might appear at different
scales.
How do we determine these scales?
•
Need a scale selection mechanism!
32
Scale Selection: Exhaustive Search
• Simple approach for scale selection but not
efficient!
33
Scale Selection: Characteristic Scale
•
•
Scale selection by finding the characteristic scale of each
feature.
This scale reveals the spatial extent of an interest point.
characteristic scale
characteristic scale
34
Scale Selection: Characteristic Scale
•
Only a subset of interest points are selected using the
characteristic scale of each feature  matching can
be simplified.
•
The size of the
circles is
related to the
scale at which
the interest
points were
selected.
35
Automatic Scale Selection
• Design a function F(x,σn) which provides some
local measure.
• Select points at which F(x,σn) is maximal over σn.
max of F(x,σn)
corresponds to
characteristic scale!
F(x,σn)
σn
T. Lindeberg, "Feature detection with automatic scale selection“, International
Journal of Computer Vision, vol. 30, no. 2, pp 77-116, 1998.
36
Lindeberg et al, 1996
Slide from Tinne Tuytelaars
Automatic Scale Selection
•
Using characteristic
scale, the spatial
extent of interest points
becomes covariant to
scale transformations.
•
The ratio σ1/σ2 reveals
the scale factor
between the images.
σ1
σ2
σ1/σ2 = 2.5
45
How to Choose F(x,σn) ?
• What local measure F(x,σn) should we use?
− Should be rotation invariant
− Should have one stable sharp peak
46
How to Choose F(x,σn) ?
• Typically, F(x,σn) is defined using derivatives, e.g.:
Square gradient :  2 ( L2x ( x,  )  L2y ( x,  ))
LoG : |  2 ( Lxx ( x,  )  Lyy ( x,  )) |
DoG :| I ( x)* G ( n 1 )  I ( x)* G ( n ) |
Harris function : det( AW )   trace2 ( AW )
• LoG yielded best results in an evaluation study; DoG was
second best.
C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point
Detectors", International Journal of Computer Vision, 37(2), pp. 151-172,
2000.
47
How to Choose F(x,σn) ?
• Let’s see how LoG responds at blobs …
48
Recall: Edge Detection Using 1st Derivative
f
d
g
dx
d
f
g
dx
edge
derivative
of Gaussian
edge = maximum
of derivative
49
Recall: Edge Detection Using 2nd Derivative
f
d2
g
2
dx
d2
f 2g
dx
edge
2nd derivative
of Gaussian
(Laplacian)
edge = zero
crossing
of 2nd derivative
50
LoG’s Response at a Blob
• Blob = superposition of two edges
(blobs of different spatial extent)
max
(abs value)
The magnitude of the LoG response will achieve a maximum at
the center of the blob, provided the scale of the LoG is “matched”
to the scale of the blob (e.g, its spatial extent)
51
How to Find Spatial Extent of a Blob Using LoG?
• Convolve blobs with LoG filters at several scales and find
maximum response across scales.
• Problem: LoG response decays as scale increases.
Blob
increasing σ
Why does this happen?
52
Gaussian Derivative Response
• The response of the Gaussian derivative to a
perfect step edge decreases as σ increases.
1
 2
53
Normalize LoG Response
• To keep response the same (i.e., scale-invariant),
must multiply Gaussian derivative by σ
• LoG is the second Gaussian derivative, so it must
be multiplied by σ2
54
Normalize LoG Response
Blob
Un-normalized LoG response
Scale-normalized LoG response ( σ2 x LoG)
max
55
Spatial Extent of Circle
• At what scale does the LoG achieve a maximum
response for a binary circle of radius r?
r
image
2 (characteristic scale)
LoG response
• LoG is maximized at   r /
r/ 2
scale (σ)
56
Spatial Extent of Circle
• Spatial extent r can be determined using: r  2
characteristic scale
57
Harris-Laplace Detector
• Multi-scale Harris with scale selection
• Uses LoG maxima to find characteristic scale
σn
 LoG 
scale
y
 Harris 
x
58
Harris-Laplace Detector
(1) Find interest points at multiple scales using Harris
detector.
- Scales are chosen as follows: σn =knσ
- At each scale, choose local maxima assuming 3 x 3
window
(i.e., non-maximal suppression)
F ( x,  n )  F ( xW ,  n ) xW W
F ( x,  n )  t h
where
F ( x,  n )  det( AW )   trace 2 ( Aw )
(σD =σn, σI =γσD )
59
Harris-Laplace Detector
(2) Select points at which the normalized LoG is
maximal across scales and the maximum is
above a threshold.
σn+1
F ( x,  n )  F ( x,  n 1 )  F ( x,  n )  F ( x,  n 1 )
F ( x,  n )  t
σn
where:
σn-1
F ( x,  n ) |  2 ( Lxx ( x,  n )  Lyy ( x,  n )) |
K. Mikolajczyk and C. Schmid, “Indexing based on scale invariant interest
points“, IEEE Intl. Conference on Computer Vision, pp 525-531, 2001.
60
Example
• Interest points detected at each scale using Harris-Laplace
− Few correspondences between levels corresponding to same σ
− More correspondences between levels where ratio of σ = 2
images differ
by a scale
factor of 1.92
σ=1.2
σ=2.4
σ=1.2
σ=2.4
σ=4.8
σ=4.8
σ=9.6
σ=9.6
61
Example
(same viewpoint – change in focal length and orientation)
-More than 2000 points would have been detected without scale
selection.
-Using scale selection, 190 and 213 points were detected in the left
and right images, respectively.
62
Example
58 points are initially matched (some not correctly)
63
Example
• Reject outliers (inconsistent matches – using RANSAC)
• Left with 32 matches, all of which are correct.
•Estimated scale factor is 4:9 and rotation angle is 19°.
64
Harris-Laplace Detector
Repeatability
• Invariant to:
− Scale
− Rotation
− Translation
• Robust to:
− Illumination changes
− Limited viewpoint changes
65
Implement Harris-Laplace Using DoG
• LoG can be approximated by DoG:
G( x, y, k )  G( x, y,  )  (k  1) 22G
• Note: DoG already
incorporates the σ2 scale
normalization factor.
66
Implement Harris-Laplace Using DoG
•
Gaussian-blurred image
L ( x, y ,  )  G ( x, y ,  ) * I ( x, y )
•
Result using DoG:
DoG ( x, y,  )  L( x, y, k )  L( x, y,  )
-
=
67
Implement Harris-Laplace Using DoG
G  x, y, k 2  * I
G  x, y, k  * I
D  x, y ,   
 G  x, y, k   G  x, y,    * I
G  x, y,   * I
σn =knσ
DoG
68
Harris-Laplace using DoG
Look for local maxima
in DoG pyramid
DoG pyramid
David G. Lowe, "Distinctive image features from scale-invariant keypoints.”
Intl. Journal of Computer Vision, 60 (2), pp. 91-110, 2004.
69
Handling Affine Changes
• Similarity transformations cannot account for
perspective distortions; affine could be used for
planar surfaces.
• Similarity transform
• Affine transform
70
Handling Affine Changes
• Similarly to characteristic scale selection, detect
the characteristic shape of the local feature.
• Need to create scale space using non-uniform
Gaussian filters (i.e., Affine Gaussian Scale
Space)
71
Uniform Gaussian Scale Space
Uniform Gaussian (assuming zero mean)
L ( x, y, )  G ( x, y, )  I ( x, y )
G ( x, y ,  ) 
1
2 2
e
x2  y 2

2

1
e
2 det( )
p   1 p

2
Σ: covariance matrix
 2 0 
x
where p    and   
2
y
 
0  
Only one parameter: σ
72
Affine Gaussian Scale Space
L ( x, y,  )  G ( x, y,  )  I ( x, y)
Non-uniform Gaussian (assuming zero mean)
G ( x, y ,  ) 
1
e
2 det( )

Σ: covariance matrix
p   1 p
2
 11  12 
x
where p    and   
( 12   21 )

 y
 21  22 
Three parameters: σ11, σ12, σ22
73
How Does This Affect Harris’ 2nd Order Moment Matrix?
• Assuming uniform Gaussian:
same as AW(x, σI, σD)
w(x,y)
• Assuming non-uniform Gaussian:
σI ΣI and σD ΣD
• Not practical to compute μ for all possible parameter values!
74
Iterative Affine Adaptation (Harris Affine Detector)
• Use an iterative approach:
− Extract approximate locations and scales using the Harris-Laplace
detector.
− For each point, modify the scale and shape of its neighborhood in
an iterative fashion.
− Converges to stable points that are covariant to affine
transformations
75
Steps of Iterative Affine Adaptation
1. Detect initial locations and neighborhood using HarrisLaplace.
2. Estimate affine shape of neighborhood using 2nd order
moment matrix μ(x, σI, σD).
76
Steps of Iterative Affine Adaptation
3. Normalize (i.e., de-skew) the affine region by mapping it to
a circular one (i.e., “remove” perspective distortions).
4. Re-detect the new location and scale in the normalized
image.
5. Go to step 2 if the eigenvalues of μ(x, σI, σD) for the new
point are not equal (i.e., not yet adapted to the
characteristic shape).
77
Iterative Affine Adaptation – Examples
Initial points
Example 1
Example 2
78
Iterative Affine Adaptation – Examples
Iteration #1
Example 1
Example 2
79
Iterative Affine Adaptation – Examples
Iteration #2
Example 1
Example 2
80
Iterative Affine Adaptation – Examples
Iteration #3, #4, …
Example 1
Example 2
K. Mikolajczyk and C. Schmid, “Scale and Affine invariant interest point detectors”,
International Journal of Computer Vision, 60(1), pp. 63-86, 2004.
http://www.robots.ox.ac.uk/~vgg/research/affine/
81
Iterative Affine Adaptation – Examples
82
De-Skewing
• Consider a point xL with 2nd order matrix ML
xL
• The de-skewing transformation is defined as
follows:
83
De-Skewing Corresponding Regions
• Consider two points xL and xR which are related
through an affine transformation:
xL
xR
84
De-Skewing Corresponding Regions
Normalized regions
are related by pure
rotation R.
85
Resolving Orientation Ambiguity
•
•
Create histogram of local gradient directions in
the patch.
Smooth histogram and assign canonical
orientation at peak of smoothed histogram.
Dominant gradient
direction!
2π
0
(36 bins)
86
Resolving Orientation Ambiguity
• Resolve orientation ambiguity
Compute R
87