Enhancing Exemplar SVMs using
Part Level Transfer Regularization
1
Problem Definition:
Image Retrieval
2
Problem Definition:
Image Retrieval
query
3
Problem Definition:
Image Retrieval
query
Retrieved Images
Retrieving same category in a similar pose
Image Database
Example: bicycle facing left
query
Retrieved Images
4
A Candidate Solution:
Exemplar SVM (E-SVM)
[Malisiewicz’11]
[Shrivastava’11]
Training a SVM with a single positive and many negative samples
Linear SVMs
over
HoG features
[Dalal &Triggs’05],
[Felzenszwalb’08]
Exemplar SVM
5
A Candidate Solution:
Exemplar SVM (E-SVM)
Training a SVM with a single positive and many negative samples
Linear SVMs
over
HoG features
[Dalal &Triggs’05],
[Felzenszwalb’08]
Exemplar SVM
Image Database
Retrieval via sliding window search on the image database
6
A Candidate Solution:
Exemplar SVM (E-SVM)
Training a SVM with a single positive and many negative samples
Linear SVMs
over
HoG features
[Dalal &Triggs’05],
[Felzenszwalb’08]
Exemplar SVM
Image Database
Retrieval via sliding window search on the image database
Retrieved Images
7
Framework:
Enhanced Exemplar SVM (EE-SVM)
positive sample
Train E-SVM
over
HoG features
negative samples
Previously
Trained
Classifiers
Exemplar SVM
Part-Level
Transfer
Enhanced E-SVM
8
Benefit:
Enhanced Exemplar SVM (EE-SVM)
Exemplar SVM
Subwindow
Retrieval
Query Image
Retrieved
Subwindows
Image
Database
Retrieved
Subwindows
Subwindow
Retrieval
Enhanced E-SVM
9
Overview
• Transfer Learning in Computer Vision
– Classification & Detection
• Enhanced Exemplar SVM
• Feature Augmentation vs Transfer
• Results & Discussion
10
Transfer Learning in Computer Vision
Learning new classes by building upon previously learned classes.
• Image Classification
– Adaptive SVMs,
– Transfer from Multiple Models,
– Adaptive Multiple Kernel Learning
• Object Detection
– Rigid Transfer
– Flexible Transfer
[Yang et al. ICDM’07]
[Tommasi et al. BMVC’09]
[Tommasi et al. CVPR’10]
[Luo et al. ICCV’11]
[Duan et al. CVPR’10]
[Stark et al. ICCV’09]
[Aytar and Zisserman ICCV’11]
[Gao et al. ECCV’12]
13
Transfer Learning for Detection
Fixed Sized
Transfer
• Rigid Transfer [Aytar and Zisserman ICCV’11]
– Transfer between fixed sized templates
– Good performance, especially for smaller number of training samples.
– Hard to find visually similar detectors with same aspect ratio and size.
Flexible
Transfer
• Flexible Transfer
–
–
–
–
Transfer between different sized templates.
Transferring shape features [Stark et al. ICCV’09]
Deformable Transfer [Aytar and Zisserman ICCV’11]
Transfer via Structured Priors [Gao et al. ECCV’12]
14
Overview
• Transfer Learning in Computer Vision
– Classification & Detection
• Enhanced Exemplar SVM
• Feature Augmentation vs Transfer
• Results & Discussion
15
Framework:
Enhanced Exemplar SVM (EE-SVM)
Train
E-SVM
Query
Part-Level
Transfer
Enhanced E-SVM
Previously Trained
Classifiers
Exemplar SVM
16
Framework:
Part-Level Transfer Regularization
Exemplar
ui SVM
17
Parameters:
Part-Level Transfer Regularization
close to construction from ui’s
ui
close to E-SVM
18
Framework:
Matching Classifier Patches
Exemplar SVM
ui
Previously Learned Classifiers
19
Why is it beneficial?
Part-Level Transfer Regularization
• Part level transfer is beneficial because…
– parts can be relocated (deformation),
– the possibility of finding a good match for transfer increases when we
look at smaller classifier patches.
• Advantages of transferring parts from well trained classifiers:
– Better background suppression and discriminativity due to well
trained source classifiers.
– Better handling of local variations since source classifiers are trained
on many positive samples.
• No additional cost on runtime
20
Where is it beneficial?
Part-Level Transfer Regularization
• Unusual Poses
• Composition of Objects [Visual Phrases - Sadeghi CVPR’11]
21
PASCAL 2007:
Results - Left Facing Horse
query
Enhanced E-SVM
E-SVM
22
PASCAL 2007:
Results - Left Facing Bicycle
query
Enhanced E-SVM
E-SVM
23
PASCAL 2007:
Visual Phrase – Riding Horse
query
Enhanced E-SVM
E-SVM
24
ImageNet:
Unusual Pose - Bicycle
query
Enhanced E-SVM
E-SVM
25
Overview
• Transfer Learning in Computer Vision
– Classification & Detection
• Enhanced Exemplar SVM
• Feature Augmentation vs Transfer
• Results & Discussion
27
Implementation:
Transfer vs. Feature Augmentation
....
Transfer Regularization
is equivalent to learning
.
0.2 with augmented
0.7
0.1 features.
“normal” SVM
.
.
29
Implications:
Transfer vs. Feature Augmentation
• This equivalence is not specific to Exemplar
SVMs.
• Transfer regularization can be implemented as
feature augmentation.
• Transfer regularization can be efficiently
solved using standard SVM packages.
30
Overview
• Transfer Learning in Computer Vision
– Classification & Detection
• Enhanced Exemplar SVM
• Feature Augmentation vs Transfer
• Results & Discussion
31
PASCAL 2007:
Quantitative Results
32
ImageNet:
Quantitative Results
• Three queries are evaluated for each of the five classes.
• Precisions at top 5, 10, 50 and 100 are reported.
33
EE-SVM
E-SVM
Query
Handling Occlusions
34
EE-SVM
E-SVM
Query
Handling Truncation
35
Conclusions
• Boosted the performance of E-SVM which incurs no
additional cost on runtime.
• Presented the equivalence between Transfer
regularization and feature augmentation.
• Showed the benefit for unusual poses and visual
phrases.
• Handling truncation and occlusion.
36