Heuristic Pre-Clustering Relevance Feedback
in Attention-Based Image Retrieval
Wan-Ting Su, Wen-Sheng Chu
and Jenn-Jier James Lien
Speaker: Wen-Sheng Chu
Robotics Lab. CSIE NCKU
System Interface
Query Image
Positive
Feedback
Negative
Feedback
Heuristic Pre-Clustering View
User can change the
positive group number
on his/her own
User can revise the
clustering results
manually
Result View
Robotics Lab, CSIE NCKU
System Overview
Offline Module : Attention-Based Image Retrieval
Low-Low
Subband
Wavelet
Transformation
Image
Database
Query
Image
No
END
User
Feedback?
Attended View
Extraction
Feature Extraction
from Attended View
Best
Matches
Ranking by
Euclidean Distance
Yes
Ranking by GBDA
Learning
PCA
Heuristic
Pre-clustering
VQ
User
Re-clustering
Online Module : Heuristic Pre-Clustering Relevance Feedback
Robotics Lab, CSIE NCKU
Wavelet and
Attended View Extraction
• To reduce the computational cost
• Contrast extraction is applied to the wavelet
coefficient in the LL-subband.
attention center
Gaussian distance
Ci , j   d ( pi , j , q)
q
contrast value of
pixel p at image
location (i, j)
Got
saliency
map!
neighborhood of
pixel (i, j)
1

x

0

CM


 y0  1
CM


N 1 M 1
 C
j 0 i 0
M 1 N 1
 C
i 0 j 0
M 1 N 1
 C
i 0 j 0
Robotics Lab, CSIE NCKU
i, j
i, j
i
i, j
j
System Overview
Offline Module : Attention-Based Image Retrieval
Low-Low
Subband
Wavelet
Transformation
Image
Database
Query
Image
No
END
User
Feedback?
Attended View
Extraction
Feature Extraction
from Attended View
Best
Matches
Ranking by
Euclidean Distance
Yes
Ranking by GBDA
Learning
PCA
Heuristic
Pre-clustering
VQ
User
Re-clustering
Online Module : Heuristic Pre-Clustering Relevance Feedback
Robotics Lab, CSIE NCKU
Visual Features Extraction
• Table1. 32 low-level visual features
Features
Dimension
Color mean, standard deviation and skew
in HSV space
9
Standard deviation of the wavelet
coefficients in 10 pyramid de-correlated
sub-bands
10
13 statistical elements extracted from the
edge map such as max fill time, max fork
count, etc.
13
Robotics Lab, CSIE NCKU
System Overview
Offline Module : Attention-Based Image Retrieval
Low-Low
Subband
Wavelet
Transformation
Image
Database
Attended View
Extraction
Feature Extraction
from Attended View
Got features!
Query
Image
No
END
User
Feedback?
Best
Matches
Yes
Ranking by GBDA
Learning
PCA
Heuristic
Pre-clustering
VQ
Ranking by
Euclidean Distance
User
Re-clustering
Online Module : Heuristic Pre-Clustering Relevance Feedback
Robotics Lab, CSIE NCKU
Pre-Clustering
• Principal Component Analysis (PCA)
+
• Vector Quantization algorithm (VQ)
Robotics Lab, CSIE NCKU
User Re-clustering
User Re-clustering
System Pre-clustering Result
Robotics Lab, CSIE NCKU
User Re-clustering Result
System Overview
Offline Module : Attention-Based Image Retrieval
Low-Low
Subband
Wavelet
Transformation
Image
Database
Query
Image
No
END
User
Feedback?
Attended View
Extraction
Feature Extraction
from Attended View
Best
Matches
Ranking by
Euclidean Distance
Yes
Ranking by GBDA
Learning
PCA
Heuristic
Pre-clustering
VQ
User
Re-clustering
Online Module : Heuristic Pre-Clustering Relevance Feedback
Robotics Lab, CSIE NCKU
Re-weighting Scheme
• Group-Based Discriminant Analysis (GBDA)
• Multiple positive and multiple negative classes
• Clustering each positive class
• Scattering the negative example away from each
positive class
Negative Samples
Bouquets of
Flowers
Single Flower
Positive Samples
Robotics Lab, CSIE NCKU
GBDA
W T S PNW
W  arg max T
W SW W
W
Sw : the sum of the within-class scatter matrix of the
positive groups
c
S w   Si
i 1
Si   xC (x  mi )(x  mi )T
i
mi : the mean of the ith
positive class Ci
c: the number of
positive groups
SPN is the sum of between-class scatter matrices of
positive-to-negative
c
S PN   S Ni
i 1
S Ni   yD (y  mi )(y  mi )T
D : a set of negative examples
Robotics Lab, CSIE NCKU
Experiment Result
(1)
• COREL image database
• QS2: 1000 images from 10 selected categories
• Each of 10 categories contains 100 images and is
used as queries.
Table 1. Image Categories in Query Set 2
1. Sunset
2. Flower
3. Car
4. Ape
5. Mountain
6. Penguin
7. Tiger
8. Bird
9. Horse
10. Building
precision
relevantimages retrievedin topN returns
N
Robotics Lab, CSIE NCKU
Experiment Result
60.00%
(2)
Attention-Based System
Global
55.00%
Precision
50.00%
45.00%
40.00%
35.00%
30.00%
25.00%
20.00%
10
20
30
40
50
60
Scope
Robotics Lab, CSIE NCKU
70
80
90
100
Experiment Result
80.00%
(3)
Attention-Based System
Global
70.00%
Precision
60.00%
50.00%
40.00%
30.00%
20.00%
10.00%
0.00%
1
2
3
4
5
6
Category ID
Robotics Lab, CSIE NCKU
7
8
9
10
Experimental Results
Query Image
First-time
retrieval
results
Precision = 5/10
Precision = 7/20
Robotics Lab, CSIE NCKU
(4)
Experimental Results
First-time
feedback
results
Precision = 8/10
Precision = 17/20
Robotics Lab, CSIE NCKU
(5)
Experimental Results
Second-time
feedback
results
Precision = 10/10
Precision = 20/20
Robotics Lab, CSIE NCKU
(6)
Conclusion
• The major work in this study is integrating
attention-based image retrieval with the relevance
feedback algorithm using multiple positive and
negative groups.
• The system guides the user in clustering positive
feedbacks by providing heuristic pre-clustering
results. Then the user can revise the clusters
manually.
Robotics Lab, CSIE NCKU
Experiment Result
- Video Demo
Robotics Lab, CSIE NCKU