Special Topic on Image Retrieval
Local Feature Matching Verification
Geometric Verification
• Motivation
– Remove false matches by checking geometric
consistency
Red line: geometric consistent match
Blue line: geometric inconsistent match
2
Global Verification: RANSAC
• Take RANSAC as an example
– Check geometric consistency from matched feature pairs.
Random
sampling
 x   m1
   
 y   m3
m2   u   t1 
    
m4   v   t 2 
Local Geometric-Verification
• Locally nearest neighbors （Video Goole, cvpr’03）
– Matched regions should have a similar spatial layout.
– For each match define its search area
– Region in the search area that also matches casts a vote for
the image
– Reject matches with no support
• Drawback
– Sensitive to clutter
Hamming Embedding (ECCV’08)
• Introduced as an extension of BOV [Jegou 08]
– Combination of
– A partitioning technique (k-means)
– A binary code that refine the descriptor
• Representation of a descriptor x
– Vector-quantized to q(x) as in standard BOV
– short binary vector b(x) for an additional localization in the Voronoi cell
• Two descriptors x and y match iif
Hamming Embedding
• Binary signature generation
– Off-line learning
• Random matrix generation
• Descriptor projection and assignment
• Median values of projected descriptors
– On-line binarization
• Quantization assignment
• Descriptor projection
• Computing the signature:
Local Geometric-Verification
• Bundled feature (CVPR’09)
– Group local features in local MSER region.
– Increase discriminative power of visual words.
– Allowed to have large overlap error.
•
Bundle comparison:
– Mm(q; p): number of common visual words between two bundles
– Mg(q; p): inconsistency of geometric order in x- and y- direction.
•
Drawbacks: Infeasible for rotated bundles.
Local Geometric-Verification
Bundled feature (CVPR’09)
# of shared
– Visual words are bundled in MSER regions.
visual words
– Spatial consistency for bundled features is utilized to weight visual
words.
M (q; p)  M m (q; p)   M g (q; p)
v  vtf idf  M (q, p )
– Great performance for partial-dup detection in over 1 M database
Spatial
– Drawbacks: Infeasible for rotated bundles.
consistency
Z. Wu, J. Sun, and Q. Ke, “Bundling Features for Large Scale Partial-Duplicate Web Image Search,” CVPR 09
Global Verification: RANSAC

RANSAC: remove outliers by inlier classification



Assumption of RANdom SAmple Consensus (RANSAC)



Inliers: true matched features
Outliers: false matched features
The original data consists of inliers and outliers.
A subset of inliers can estimate a model to optimally explain the inliers.
Estimate the affine transformation by RANSAC
 x   m1
   
 y   m3

Procedure: Iteratively select a random subset as hypothetical inliers
1.
2.
3.
4.

m2   u   t1 
    
m4   v   t 2 
A model is fitted to the hypothetical inliers.
All other data are tested against the fitted model for inlier classification.
The model is re-estimated from all hypothetical inliers.
The model is evaluated by estimating the error of the inliers relative to the model.
Drawbacks: Computationally expensive, not scalable
Fischler, et al., RANdom SAmple Consensus: a paradigm for model fitting with applications to image analysis and automated
cartography, Comm. of the ACM, 24:381-395, 1981
Spatial Coding for Geometric
Verification (ACM MM’ 10)
• Motivation
– Encode local features’ relative positions into compact binary maps
– Check spatial consistency of local matches for geometric verification
• Spatial coding maps
– Relative spatial positions between local features.
– Very efficient and high precision
Zhou & Tian, Spatial Coding for large scale
partial-duplicate image search. ACM Multimedia 2010.
1

1
Xmap  
1

0

1

0
Ymap  
0

0

0 0 1

1 0 1
1 1 1

0 0 1
1 1 1

1 1 1
0 1 1

0 0 1
Spatial Map Generation

In previous case, each quadrant has one part

Consider each quadrant is uniformly divided into two parts.
=
Rotate 45 degree
counterclockwise
11
Spatial Map Generation

Generalized spatial map: GX and GY

Each quadrant is uniformly divided into r parts.
k=0

k=r-1
GY
X-map
Y-map
X-map
Y-map
X-map
Y-map
k 
2r
k=1
…
GX
…
12
Generalized Spatial Coding

Spatial coding maps:



Each quadrant uniformly divided into r parts.
Decompose the division into r sub-division.
Rotate each sub-division to align the axis.
New feature locations after rotation ：
 xi k   cos( )  sin( )   xi 
 
 
 y k   sin( ) cos( )   yi 
 
 i 
k 

, k  0,1,2,, r  1
2r
Generalized spatial maps：
k
k

0 if xi  x j
GX (i, j , k )  
k
k
1
if
x

x

i
j

k
k

0 if yi  y j
GY (i, j , k )  
k
k
1
if
y

y

i
j

13
Spatial Verification

Verification with spatial maps GX and GY

Compare the spatial maps of matched features:
Vx (i, j , k )  GX q (i, j , k )  GX m (i, j, k ) Vx: inconsistent degree in X-map
V y (i, j , k )  GYq (i, j , k )  GYm (i, j , k ) Vy: inconsistent degree in Y-map


k=0, …, r-1; i, j=1, …, N; N: number of matched features
Find and delete the most inconsistent matched pair,
recursively:
r 1 N
S x (i)  Vx (i, j, k )
i *  arg max S x (i )
S y (i)  Vy (i, j, k )
j *  arg max S y ( j )
k 0 j 1
r 1 N
k 0 j 1
i
Identify i*
and remove
j
14
5
y
1
2
1
2
3
3
4
4
5
x
1

1
Vx  Xq  Xm   1

0

1
1

0
Vy  Yq  Ym   0

0

0
0 0 1 0 1
 
1 0 1 0 1
1 1 1 1  1
 
0 0 1 0 0
 
1 0 1 1 0
1 1 1 1  1
 
1 1 1 1  0
0 1 1 1   0
 
0 0 1 1  0
 
0 0 0 1  1
0 0 1 1  0
 
1 0 1 1  0
1 1 1 1   0
 
0 0 1 1  0
 
0 0 0 1  1
1 1 1 0  0
 
1 1 1 0  0
0 1 1 0   0
 
0 0 1 0  0
 
1 1 1 1 1
0 0 0 1

0 0 0 1
0 0 0 0 ,

0 0 0 1

1 0 1 0
0 0 0 1

0 0 0 1
0 0 0 1 ,

0 0 0 1

1 1 1 0
1
 
1
Sx   0 
 
1
 
 3
1
 
1
Sy   1 
 
1
15 
 4
Geometric Verification with Coding
Maps
5
1
y
2
1
2
3
3
4
4
x
1

1
Vx  Xq  Xm   1

0

1
1

0
Vy  Yq  Ym   0

0

0
5
0 0 1 0 1
 
1 0 1 0 1
1 1 1 1  1
 
0 0 1 0 0
 
1 0 1 1 0
1 1 1 1  1
 
1 1 1 1  0
0 1 1 1   0
 
0 0 1 1  0
 
0 0 0 1  1
0 0 1 1  0 0 0 0 1 

 
1 0 1 1  0 0 0 0 1 
1 1 1 1   0 0 0 0 0 ,

 
0 0 1 1  0 0 0 0 1 

 
0 0 0 1  1 1 0 1 0 
1 1 1 0  0 0 0 0 1

 
1 1 1 0  0 0 0 0 1
0 1 1 0    0 0 0 0 1 ,

 


0
0
0
0
1

0 0 1 0

 
1 1 1 1 1 1 1 1 0
SUM
1
 
1
Sx   0 
 
1
 
 3
1
 
1
Sy   1 
 
1
 
 4
16
Image Plane Division (TOMCCAP’ 10)
4
4
3
2
3
2
1
4
2
1
1
5
5
(a)
4
5
(b)
(c)
4
4
3
3
2
3
2
2
5
1
5
1
(d)
3
5
1
(e)
(f)
17
Geometric Square Coding
• Coordinate adjustment
4
4
 x j   cos(i )  sin( i )   x j 
3 i, j  M
 (i )2 3 




,
0
 y   sin( i ) cos(i )   y j  2
 
1 j 
5
(i )
5
1
• Square coding map


(i )
(i )
(i )
(i )

1 if max x j  xi , y j  yi  si
Smap(i, j )  

otherwise
0
Generalized map： GS (i, j ) 
max( x j
(i )
 xi , y j
(i )
si
(i )
 yi )
(i )
18
Geometric Fan Coding
• Fan coding maps
(i )
(i )

0
if
x

x

j
i
Hmap(i, j )  
(i )
(i )
1
if
x

x

j
i

i
i

0 if y j  yi
Vmap(i, j )  
i
i
1
if
y

y

j
i

• Coordinate adjustment
 x j (i , k )   cos(i ( k ) )  sin( i ( k ) )   x j 
 (i , k )   
 
(
k
)
(
k
)

  yj 
y

sin(

)
cos(

)
j
i
i
  

 
• Generalized coding maps
(i , k )
(i , k )

0
if
x

x

j
i
GH (i, j, k )  
(i , k )
(i , k )
1
if
x

x

j
i

(i , k )
(i , k )

 yi
0 if y j
GV (i, j , k )  
(i , k )
(i , k )
1
if
y

y

j
i

19
Geometric Verification
• Compare the fan coding maps of matched features:
• Inconsistency measurement from geometric fan coding:
• Inconsistency measurement from geometric square coding:
• Inconsistency matrix: