The Correspondence Problem
and “Interest Point” Detection
Václav Hlaváč
Center for Machine Perception
Czech Technical University Prague
hlavac@fel.cvut.cz
Courtesy
Martin Urban (the talk is based on his
presentation) and Jana Kostkova, Jan Kybic,
Jiri Matas, Radim Sara
Lecture Overview
1.
The Correspondence Problem (CP) lies at the core of a number of
computer vision problems:
•
•
•
•
•
•
2.
3.
4.
5.
tracking (correspondence in consecutive frames
narrow-baseline stereo
wide-baseline stereo
egomotion (camera motion) estimation
motion segmentation in sequences with a moving camera
recognition, categorisation, …
Demo of typical applications
The CP is commonly solved by robust matching of “Interest Points”
“Interest Points” are regions with distinguishing property, allowing
there detection in a viewpoint and illumination invariant manner
Harris “Corner” (HC) Detection Algorithm
•
•
HC are rotation and translation invariant interest points
HC are interest points most commonly used in tracking and stereo
2
What it is “The correspondence problem”?
A
B
Ex. result: x-shift
“Ideal” solution:
a pixel to pixel mapping from A to B (B ⋃ NULL)
3
Applications: 3D Reconstruction
3D Reconstruction
Camera motion tracking ⇒ image stabilization
original
original
stabilized
stabilized
Camera motion tracking ⇒ 3D animation
Motion keying / segmentation
input sequence
Background sequence
Foreground sequence
Motion Detection
Input:
Output:
Medical imaging – image registration
from the atlas
before registration
after
test slice
deform. field
The correspondence problem
A
“Ideal” solution:
B
a pixel to pixel mapping from A to B
How to do it?
11
The correspondence problem
A
Intuitive approach:
For ∀ pxl ∈ A, find a pxl ∈ B
with the most similar neighbourhood.
B
Problems: - How to measure similarity of image patches?
- undistinguishable regions (e.g. texture-less)
- not surjective map (onto) due to occlusions
- not bijective map (one to one) due to scale changes
- huge data
⇒ very hard / impossible to recover the mapping by direct minimisation
12
The correspondence problem
Conclusion:
It is very hard / impossible to recover dense pxl 2 pxl mapping between
two images.
Solution:
1. Recover the correspondence relation just between several well
distinguished image features (interest points / corners / Harris points).
2. Estimate multi-view transformation (e.g. epipolar geometry, camera
motion).
3. Having epipolar geometry, try to find dense correspondences.
13
Corner Detection:
Introduction
undistinguished patches:
distinguished patches:
Corner detector detects points with distinguished neighbourhood(*) well
suited for matching verification.
14
Corner Detection:
Introduction
+
-
>0
Demo of a point + with well distinguished neighbourhood.
15
Corner Detection:
Introduction
+
-
>0
Demo of a point + with well distinguished neighbourhood.
16
Corner Detection:
Introduction
+
-
>0
Demo of a point + with well distinguished neighbourhood.
17
Corner Detection:
Introduction
+
-
>0
Demo of a point + with well distinguished neighbourhood.
18
Corner Detection:
Introduction
+
-
>0
Demo of a point + with well distinguished neighbourhood.
19
Corner Detection:
Introduction
+
-
>0
Demo of a point + with well distinguished neighbourhood.
20
Corner Detection:
Introduction
+
-
>0
Demo of a point + with well distinguished neighbourhood.
21
Corner Detection:
Introduction
+
-
>0
Demo of a point + with well distinguished neighbourhood.
22
Corner Detection:
Introduction
Example of detected points
23
Corner Detection:
undistinguished patches:
Basic principle
distinguished patches:
Image gradients ∇I(x,y) of undist. patches are (0,0) or have only one
principle component.
Image gradients ∇I(x,y) of dist. patches have two principle components.
⇒
rank ( ∑ ∇I(x,y)* ∇I(x,y) ⊤ ) = 2
24
Algorithm (C. Harris, 1988)
1. filter the image by gaussian (2x 1D convolution), sigma_d
2. compute the intensity gradients ∇I(x,y), (2x 1D conv.)
3. for each pixel and given neighbourhood, sigma_i:
- compute auto-correlation matrix
A = ∑ ∇I(x,y)* ∇I(x,y) ⊤
- and evaluate the response function R(A):
R(A) >> 0 for rank(A)=2, R(A) → 0 for rank(A)<2
4. choose the best candidates (non-max suppression and
thresholding)
25
Corner Detection:
Algorithm (R. Harris, 1988)
Harris response function R(A):
R(A) = det (A) – k*trace 2(A) ,
[lamda1,lambda2] = eig(A)
26
Corner Detection:
Algorithm (R. Harris, 1988)
Algorithm properties:
+ “invariant” to 2D image shift and rotation
+ invariant to shift in illumination
+ “invariant” to small view point changes
+ low numerical complexity
- not invariant to larger scale changes
- not invariant to high contrast changes
- not invariant to bigger view point changes
27
Corner Detection:
Algorithm (C. Harris, 1988)
Exp.: Harris points and view point change
28
Corner Detection:
Harris points versus sigma_d and sigma_i
↑
Sigma_d
Sigma_I
→
29
Corner Detection:
Application
3D camera motion tracking / 3D reconstruction
Algorithm:
1.
Corner detection
2.
Tentative correspondences
3.
Camera motion geometry estimation (e.g. by RANSAC)
- by comparing similarity of the corner neighb. in the searching window
(e.g. cross-correlation)
- finds the motion geometry and consistent correspondences
4. 3D reconstruction
- triangulation, bundle adjustment
30
Corner Detection:
Camera Tracking Application - Boujou
Input sequence
31
Corner Detection:
Camera Tracking Application - Boujou
Harris points
32
Corner Detection:
Camera Tracking Application
points consistent with 3D camera motion
33
Corner Detection:
Camera Tracking Application
3D points and 3D camera motion
34
Corner Detection:
Camera Tracking Application
3D animation
35
Corner Detection:
Camera Tracking Application - Boujou
Input sequence
36
Corner Detection:
Camera Tracking Application - Boujou
Harris points
37
Corner Detection:
Camera Tracking Application - Boujou
points consistent with 3D camera motion
38
Acknowledgements
for the presentation I copied almost done presntation
•
Martin Urban
for used demo images and software:
•
•
•
Jana Kostková, CMP - (slide 3: images, disparity map)
Radim Šára, CMP
- (slide 5: images, 3D face reconstruction demo)
Jan Kybic, CMP
- (slide 10: medical image registration)
•
•
•
2d3 – (slide 31-38: Boujou Demo, slide 6: img. sequences)
ImagineerSystems – (slide 8: MoKey Demo, slide 8: img. sequences)
The Pixel Farm – (slide 31-33: img. sequence)
39