Relative Attributes
Presenter: Shuai Zheng (Kyle)
Supervised by Philip H.S. Torr
Author: Devi Parikh (TTI-Chicago) and Kristen Grauman (UT-Austin)

What is visual attributes?
• Attributes are properties observable in images that have humandesignated names, such as ‘Orange’, ‘striped’, or
‘Furry’.

Learning Binary Attributes
• In PASCAL VOC Challenge, we learn to predict binary attributes. (e.g., dog? Or not a dog?)
O. Parkhi, A.Vedaldi C.V.Jawahar, A.Zisserman. The Truth About Cats and Dogs. ICCV 2011.
Vittorio Ferrari, Andrew Zisserman. Learning Visual Attributes. NIPS 2007.

Problems within Binary Attributes
• Given an attribute it is easy to get labelled data on AMT(Amazon Mechanical Turk).
• But, where do attributes come from? Can we find a easier way to ask more people rather than experts to tag the images?

Problems within Binary Attributes
Some tags are binary while some are relative.
Is furry
Has four-legs
Legs shorter than horses’
Mule
Has tail
Tail longer than donkeys’

Labeling data

What is relative attributes?
• Relative attribute indicates the strength of an attribute in an image with respect to other image rather than simply predicting the presence of an attribute.

Advantages of Relative Attributes
• Enhanced human-machine communication
• More informative
• Natural for humans
8

Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image

Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image

Learning Relative Attributes
For each attribute
Supervision is open

Learning Relative Attributes
Image features
Learn a scoring function that best satisfies constraints:
Learned parameters
12

Learning Relative Attributes
Max-margin learning to rank formulation
2
1
6
4
5
3
Based on [Joachims 2002]
Image Relative Attribute Score
Rank Margin
13

Learning binary attributes v.s.
Learning relative attributes

Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image

Relative Zero-shot Learning
Training: Images from S seen categories and
Descriptions of U unseen categories
Age: Hugh Clive Scarlett Jared Miley
Smiling: Miley Jared
Need not use all attributes, or all seen categories
Testing: Categorize image into one of S + U categories
16

Relative Zero-shot Learning
Can predict new classes based on their relationships to existing classes – without training images
Age: Hugh Clive
Jared Miley
Scarlett
Smiling: Miley Jared
Miley
S
J
Age
Infer image category using max-likelihood
Clive
H
17

Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image

Automatic Relative Image Description
Density
Novel image
Conventional binary description: not dense
Dense: Not dense:
19

Automatic Relative Image Description
Density
Novel image more dense than less dense than
20

Automatic Relative Image Description
Density
Novel image
C C H H H C F H H M F F I F more dense than Highways , less dense than Forests
21

Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image

Datasets
Outdoor Scene Recognition (OSR)
[Oliva 2001]
Public Figures Face (PubFig)
[Kumar 2009]
8 classes, ~2700 images, Gist
6 attributes: open, natural, etc.
8 classes, ~800 images, Gist+color
11 attributes: white, chubby, etc.
Attributes labeled at category level http://ttic.uchicago.edu/~dparikh/relative.html
23

Baselines
• Zero-shot learning
– Binary attributes:
Direct Attribute Prediction
[Lampert 2009]
– Relative attributes via furry big classifier scores
• Automatic image-description
– Binary attributes bear turtle rabbit
24

Relative Zero-shot Learning
Binary attributes
Rel. att.
(classifier)
Rel. att.(ranke r)
An attribute is more discriminative when used relatively
25

Automatic Relative Image Description
Binary (existing):
Not natural
Not open
Has perspective
Relative (ours):
More natural than insidecity
Less natural than highway
More open than street
Less open than coast
Has more perspective than highway
Has less perspective than insidecity
26

Automatic Relative Image Description
18 subjects
Test cases:
10OSR, 20 PubFig
27

Traditional Recognition
Dog Chimpanzee Tiger
28

Attributes-based Recognition
Dog
Furry
White
Chimpanzee
Black
Big
[Lampert 2009]
[Farhadi 2009]
[Kumar 2009]
[Berg 2010]
[Parikh 2010]
…
Zero-shot learning
Describing objects
Face verification
Attribute discovery
Nameable attributes
…
Tiger
Striped
Yellow
Attributes provide a mode of communication between humans and machines!
Striped
Black
White
Big
29

Conclusions and Future Work
Relative attributes learnt as ranking functions
– Natural and accurate zero-shot learning of novel concepts by relating them to existing concepts
– Precise image descriptions for human interpretation
Enhanced human-machine communication
Attributes-based recognition is an interesting direction for the future object/scenes recognition.
30

Created by Tag clouds
Cheers! – Shuai Zheng (Kyle) kylezheng04@gmail.com