Image Keypoint Detection, Description, and

Image Keypoint Detection, Description, and Matching

AIRobots Summer School 2012

Stefan Leutenegger

Example: Build a Panorama

This panorama was generated using AUTOSTITCH (freeware), available at http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html

Thursday, 5. July 12 Image Keypoint Detection, Description, and Matching

© R. Siegwart , D. Scaramuzza, M. Chli, and S. Leutenegger, ETH Zurich - ASL

2

How Do We Build a Panorama?

§   We need to match (align) images



3


§   Detect feature points in both images



4



§   Find corresponding pairs



5



§   Find corresponding pairs

§   Use these pairs to align images



6

More Motivation …

§   Image keypoints are used also for:

§   3D reconstruction

§   Motion tracking

§   Robot navigation (Visual Odometry / SLAM)

§   Object recognition

§   Image alignment (panoramas)

§   Indexing and database retrieval (e.g. Google Images)

§   …

Image Keypoint Detection, Description, and Matching


7 Thursday, 5. July 12

Quality of Keypoints

§   Challenge 1:

§   Detector repeatability

§   Challenge 2:

§   Descriptor quality bad

?

§   More Challenges:

§   Speed: Detection and description / Matching, database retrieval

§   Invariance with respect to rotation and/or scale?

§   Tolerance to brightness / contrast changes

§   Distribution in the image




Contents

§   Keypoint detection

§   The problem and example solutions (Harris, FAST)

§   Scale invariance and examples (SIFT, BRISK)

§   Rotation invariance

§   Descriptors: extraction and matching

§   SIFT

§   Binary descriptors BRIEF and BRISK

§   Evaluation framework

§   Detector repeatability

§   Descriptor similarity

§   Example evaluation




Identifying Corners

§   Shifting a window in any direction should give a large change in intensity in at least 2 directions

Thursday, 5. July 12

“flat” region:  no intensity change

“edge” :  no change along the edge direction


“corner” :  significant change in at least 2 directions


10

How Do We Implement This?

SSD ( Δ x , Δ y ) ≈ x ,

∑

y ∈ P

( I x

( x , y ) Δ x + I y

( x , y ) Δ y ) 2 )

=

[

Δ x Δ y ] M

⎡

⎢

⎣

Δ x

Δ y

⎤

⎥

⎦

§   M is the “Second Moment Matrix”

M = x ,

∑

y ∈ P

⎡

⎢

⎣

I

I x x

I

2 y

I

I x y

I

2 y

⎤

⎥

⎦

§   Since M is symmetric, we can rewrite M:

M = R T

⎡

⎢

⎣

λ

1

0 λ

0

2

⎤

⎥

⎦

R where λ

1

and λ

2

are the Eigenvalues of M.




Corner Response Function

[Harris and Stephens, AVC88]

λ

2

§   «Cornerness Function»

R

=

=

λ

1 det(

λ

2

−

M

κ (

)

λ

1

− κ

+ λ trace

2

) 2

( M ) 2

Where κ is a constant between 0.04 and 0.15.

Edge:

λ

2

>> λ

1

Corner:

λ

1

, λ

2 are both large

§   Note : this is invariant to rotation by definition.

Flat:

λ

1

≈ λ

2

≈ 0


Edge:

λ

1

>> λ

2

λ

1



FAST(-ER) Corners

[Rosten et. al., PAMI2010]

§   Test: All points in circular segment brighter/darker by T than center ( p )

§   Very fast: in the order of 100 Mega-pixel/second


13 Thursday, 5. July 12 Image Keypoint Detection, Description, and Matching

Scale Invariant Detection

§   Consider regions (e.g. circles) of different sizes around a point

§   Corresponding regions will look the same in image space, when the appropriate scale-change is applied



14

Excursion: Spatial Filters

2M+1

S xy

( x , y )

2N+1

( x , y )

Image I Filtered Image

J = F ( I )

§   E.g. an averaging filter: J ( x , y ) =

( 2 M

( r , c

∑

) ∈ S

I xy

( r , c )

+ 1 )( 2 N + 1 )

§   Equivalent to Convolution of (windowed) Kernel k with the image:

J ( x , y ) = k ( x , y ) ∗ I ( x , y )


15 Thursday, 5. July 12 Image Keypoint Detection, Description, and Matching

Blob Detection: LoG and DoG

§   Scale( t )-normalized Laplacian of

Gaussian (LoG):

§   Image smoothed by a Gaussian kernel:

L = g ( x , y , t ) ∗ I ( x , y )

§   Apply Laplacian operator:

∇

2 norm

L = t

⎛

⎜⎜

⎝

∂

∂

2 x

L

2

+

∂

2 L

∂ y 2

⎞

⎟⎟

⎠

§   Approximation: Difference of

Gaussians (DoG) – SIFT detector

[Lowe et.al., IJCV04]

Δ L = L ( x , y , t ) − L ( x , y , kt )

Micah Taylor, www.kixor.net

, accessed June 2012


Image Keypoint Detection, Description, and Matching 16 Thursday, 5. July 12

SIFT Detector (Keypoint Location + Scale)

Blur

1.

  Scale-space pyramid: subsample and blur original image

2.

 

Difference of Gaussians (DoG) pyramid: subtract successive smoothed images

3.

  Keypoints: local extrema in the DoG pyramid

DoG:

[LoweEtal,IJCV04]




Scale-Space Corners: BRISK Detector

[Leutenegger, Chli and Siegwart, ICCV11]

§   Detector

§   High-speed (FAST based)

§   Rotation and scale invariant



18

SIFT: Orientation Assignment

Find ‘orientation’ of keypoint to achieve rotation invariance

§   Sample intensities around the keypoint

§   Compute a histogram of orientations of intensity gradients

§   Peaks in histogram: dominant orientations

§   Keypoint orientation = histogram peak

§   If there are multiple candidate peaks, construct a different keypoint for each such orientation

0 2

π




SIFT Descriptor

§   Simplest descriptor: image patch

§   We want it to be tolerant/invariant to certain changes, but yet very discriminative

§   Describe gradient orientations relative to the keypoint orientation

§   Divide keypoint neighborhood into

4 × 4 regions and compute orientation histograms along 8 directions

§   SIFT descriptor: concatenation of all

4 × 4 × 8 (=128) values

§   Matching: L

2

-distance between these descriptor vectors



20

BRIEF: Binary Robust Independent Elementary

Features

[Calonder et. al, ECCV10]

§   Goal: high speed

§   Originally no scale/rotation invariance

§   Binary descriptor formation:

§   Smooth image

§   For each detected keypoint (FAST),

Sample all intensity pairs ( I

1

, I

2

)

(typically 256 in total) according to pattern around the keypoint. If I

1

< I

2

, add a 1 to the descriptor, otherwise 0.

§   Allows very fast Hamming

Distance matching (count the bits that are different)

Pattern for intensity pair samples – generated randomly



21

BRISK Descriptor

§   Pattern used to access image intensity values in a keypoint neighborhood

§   Red circles indicate the size of the smoothing kernel applied.

§   Scaled and rotated versions stored in a look-up table

§   Sampling point locations and smoothing is configurable

§   Used for orientation assignment and brightness comparisons for binary descriptor

§   Detection and descriptor speed:

≈ 10 times faster than SURF


BRISK sampling pattern


22

Evaluation Framework

[Mikolajczyk et. al., IJCV2004], [Mikolajczyk and Schmid, PAMI2005]

§   Needed:

§   image pairs for various image transformations

§   Ground truth pixel-wise correspondence (here: homographies)

Viewpoint change Viewpoint Scale and rotation JPG compression

Brightness/contrast Blur


Blur



24

Detector Repeatability & Descriptor Similarity

Detector repeatability

§   Assume k

1

and k

2

are detected keypoints in images 1 and 2

(amount: N

1

and N

2

)

§   Project k

1

into image 2 using ground truth

§   Count the number of re-detected keypoints ( #correspondences )

§   Calculate repeatability score as score =

# correspond ences min( N

1

, N

2

)

Descriptor similarity

§   For varying descriptor distance thresholds count: correct matches and false matches.

§   Calculate: and




Example Detection Evaluation



26

Example Descriptor Similarity Evaluation


Wall rotated 60° Brightness/contrast



27

Example: Influence of Rotation and Scale Invariance

Small viewpoint change (wall 1-2)


Small rotation and scale change



28

Image Keypoint Detection, Description, and

Image Keypoint Detection, Description, and Matching

∑

∑

∑

Related documents

Products

Support

Image Keypoint Detection, Description, and

Image Keypoint Detection, Description, and Matching

∑

∑

∑

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib