5400
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 12, DECEMBER 2014
Weakly Supervised Visual Dictionary Learning
by Harnessing Image Attributes
Yue Gao, Senior Member, IEEE, Rongrong Ji, Senior Member, IEEE, Wei Liu, Member, IEEE,
Qionghai Dai, Senior Member, IEEE, and Gang Hua, Senior Member, IEEE
Abstract— Bag-of-features (BoFs) representation has been
extensively applied to deal with various computer vision applications. To extract discriminative and descriptive BoF, one
important step is to learn a good dictionary to minimize the
quantization loss between local features and codewords. While
most existing visual dictionary learning approaches are engaged
with unsupervised feature quantization, the latest trend has
turned to supervised learning by harnessing the semantic labels
of images or regions. However, such labels are typically too
expensive to acquire, which restricts the scalability of supervised
dictionary learning approaches. In this paper, we propose to
leverage image attributes to weakly supervise the dictionary
learning procedure without requiring any actual labels. As a
key contribution, our approach establishes a generative hidden
Markov random field (HMRF), which models the quantized
codewords as the observed states and the image attributes
as the hidden states, respectively. Dictionary learning is then
performed by supervised grouping the observed states, where
the supervised information is stemmed from the hidden states
of the HMRF. In such a way, the proposed dictionary learning
approach incorporates the image attributes to learn a semanticpreserving BoF representation without any genuine supervision.
Experiments in large-scale image retrieval and classification tasks
corroborate that our approach significantly outperforms the
state-of-the-art unsupervised dictionary learning approaches.
Index Terms— Bag-of-features, visual dictionary, image
attribute, weakly supervised learning, hidden Markov random
field, image classification, image search.
I. I NTRODUCTION
A
CCOMPANYING with the popularity of local features in
the recent decade, Bag-of-Features (BoF) representation
Manuscript received August 4, 2013; revised January 31, 2014, June 12,
2014, and August 1, 2014; accepted August 26, 2014. Date of publication
October 29, 2014; date of current version November 6, 2014. This work was
supported in part by the National Natural Science Foundation of China under
Grant 61422210 and Grant 61373076, in part by the Fundamental Research
Funds for the Central Universities under Grant 2013121026, and in part by the
985 Project through Xiamen University, Xiamen, China. The work of W. Liu
was supported by the Josef Raviv Memorial Post-Doctoral Fellowship. The
associate editor coordinating the review of this manuscript and approving it
for publication was Prof. Richard J. Radke. (Corresponding author: R. Ji.)
R. Ji is with the Department of Cognitive Science, School of Information
Science and Engineering, Xiamen University, Xiamen 361005, China (e-mail:
rrji@xmu.edu.cn).
Y. Gao and Q. Dai are with the Tsinghua National Laboratory for
Information Science and Technology, Department of Automation, Tsinghua
University, Beijing 100084, China.
W. Liu is with the IBM T. J. Watson Research Center, Yorktown Heights,
NY 10598 USA.
G. Hua is with the Department of Computer Science, Stevens Institute of
Technology, Hoboken, NJ 07030 USA.
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIP.2014.2364536
has been widely advocated in various computer vision
applications, e.g., visual search [11], [14], [20], object
recognition [18], [26], image parsing and scene understanding [19], [34]. Its core idea is using a visual dictionary to
encode (a.k.a. quantize) a set of local features extracted from
a given image into a BoF histogram. This representation
enjoys good robustness against photographic variations caused
by changing backgrounds, viewpoints, illuminations and
occlusions.
Visual vocabulary, or so-called visual codebook, serves as a
fundamental component in extracting the BoF representation.
It is used to encode each local feature to its best matched
word(s) [19], [20]. This coding (or quantization) procedure
will inevitably introduce information loss, which incurs a
performance drop from the raw local features to the BoF
histogram. Extensive research efforts have been devoted to
reducing such information loss [17], [20], [21], [33], [34].
One solution is to improve the feature coding (quantization)
stage, i.e. assigning one local feature to multiple codewords,
for example soft quantization [17], [20] and sparse coding [18], or alternatively, embedding the spatial layouts of local
features into quantization, for example spatial pyramid matching [34], feature bundling [33], averaged pooling [21] and max
pooling [18]. Another solution is to directly optimize the dictionary learning procedure, which aims to minimize the quantization loss of individual BoFs [8], [23], [24]. Our work falls
into the latter solution, which combines the learned dictionary
with cutting-edge feature coding designs [17], [18], [20], [21],
[33], [34] to jointly compensate the quantization loss.
Most existing methods for visual dictionary learning are
unsupervised, which focus on minimizing the quantization
errors during coding each local feature to its adjacent
codeword(s) [5], [11], [13], [17], [18], [20]. However,
in many visual search and recognition applications, the
resulting BoFs are processed as a whole in the subsequent
steps, for instance similarity ranking or classifier training.
Such applications emphasize more on the semantic-level
discriminability of the generated BoF histograms, rather
than minimizing quantization loss of individual local
features.
Therefore, a natural inspiration is to use semantic cues
like labels of regions or images to supervise the dictionary
learning process. Inspirations also originate from the human
visual system. In that case, IT cortex that is in charge of the
visual recognition function has shown stimuli feedbacks to
the V1 to V4 cortices [22], which generate the local visual
representation based on the visual stimuli.
1057-7149 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
GAO et al.: WEAKLY SUPERVISED VISUAL DICTIONARY LEARNING
Fig. 1.
5401
The proposed weakly supervised dictionary learning framework harnessing image attributes.
The labels of supervised dictionary learning are typically on
the image or the region level [8], [12], [15], [23], [24].1 Such
labels are typically leveraged to refine the similarity metric
applied to local features, or alternatively to optimize the quantization procedure. Although significant performance gains
have been reported [8], [12], [15], [18], [23], [24], [26], this
kind of supervision does not come for free. On one hand,
manually labeling images is labor intensive. On the other hand,
image tags crawled from the Internet are always noisy [27].
Moreover, the potential of the existing supervised dictionary learning approaches is restricted by their limitations
in scalability and robustness. This hinders the proliferation
of such approaches to many scenarios such as million-scale
image search, city-scale location recognition, and hyperspectral image classification.
Facilitated by the recent progress on the image
attributes [28]–[32], is it possible to exploit such cheaplyavailable supervision to “supervise” the dictionary learning
without actual labels? Image attributes reveal the possibilities
of visual cues like “Round”, “Head”, “Torso”, “Label”,
“Feather”, etc., which are acquired from other training
sources or even from different domains.2 Importantly,
attributes-based supervision enjoys good scalability and thus
enables large-scale applications like image classification,
clustering and search.
It is not guaranteed that the automatic attribute extraction
is fully accurate, which necessitates the special treatment
when being applied to supervise dictionary learning. To this
end, we propose a weakly supervised dictionary learning
approach based on a Hidden Markov Random Field (HMRF)
model. Different from the existing dictionary learning
methods [5], [11], [13], [17], [18], [20], we model the image
1 Pairwise correspondences of local feature can be also used as supervision,
which are typically derived from wide/narrow baselines matching or nearduplicate image search. This kind of labels is out of the scope of this paper.
2 For example, we use the Classemes attributes [31] in our experiments,
which were trained on the web images crawled from the Bing search engine.
attributes as the hidden nodes in HMRF and the quantized
codewords as the observation nodes. Then, the supervised
dictionary learning is conducted by iteratively optimizing the
quantization process and the attribute-to-codeword mappings.
In principle, the HMRF model seeks an optimal tradeoff
between minimizing the quantization error and fitting the
attribute responses of the BoF histograms produced. We further show that several existing unsupervised and supervised
dictionary learning methods can be explained under the proposed dictionary learning framework.
Fig. 1 outlines the proposed framework, which consists of
three interdependent components:
• Attributes-based feature coding step integrates attributes
extracted from a given image into an objective function
of feature coding by using the proposed attribute-tocodeword mapping.
• Weakly supervised dictionary learning step learns a visual
dictionary by using a constrained local feature clustering. This component can be interpreted as an attributesaware “Maximization” operation, while the previous
feature coding can be interpreted as an “Expectation”
operation.
• Attribute-to-codeword mapping step maps non-zero
attributes to particular codewords. The mapping is based
on the co-occurrence statistics between attributes and
words, which is updated after each iteration of visual
dictionary learning.
The main contributions of this paper are three-fold:
• We propose to model dictionary learning from a probabilistic aspect, i.e., using a generative Hidden Markov
Random Field. It enables us to explicitly model the supervised dictionary learning procedure under an ExpectationMaximization formulation.
• We propose to exploit the cheaply available semantic
attributes extracted from images to supervise the dictionary learning, which produces semantic-preserving BoF
descriptors without any explicit labels.
5402
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 12, DECEMBER 2014
•
by minimizing mutual information loss of the codes generated
from the training set. Moosmann et al. [23] proposed an
ERC-Forest to incorporate semantic labels to build indexing
trees. The works presented in [9] and [12] refined the initial
codewords to derive class or image specific dictionaries for
image categorization. To a certain degree, topic decomposition
including pLSA [40], [41] and LDA [42] can also be regarded
as supervised dictionary learning, which learns a topic-level
abstraction from codewords and yields a new image signature.
It is worth to note that, supervised vector quantization has been
widely studied in the area of data compression, with wellknown methods like self-organizing maps [38], regression lost
minimization [39], etc.
We demonstrate the generalization capability of the proposed approach, i.e.,. several existing dictionary learning
and feature coding schemes can be explained using the
proposed formulation.
The rest of this paper is organized as follows: Section II
reviews the related work in unsupervised and supervised dictionary learning. Section III introduces our proposed weakly
supervised dictionary learning framework. Section IV presents
the experiments with comparisons to the state-of-the-arts in
image search and image classification tasks. Section V concludes the paper and discusses our future work.
II. R ELATED W ORK
A. Unsupervised Dictionary Learning
Visual dictionary learning typically resorts to quantizing
local features, i.e., subdividing the local feature space into
disjoint partitions, each of which corresponds to a codeword.
To this end, many vector quantization schemes have been
introduced, for example, K-Means [14], Hierarchical K-Means
(a.k.a. Vocabulary Tree) [11], Approximate K-Means [13], and
their variations [6], [13], [17]. Based on the learned visual
dictionary, local features extracted from a given image are
represented as a BoF histogram, in which each codeword is
associated with a term frequency that counts how many local
features of this image fall into the corresponding partition in
the local feature space.
The most critical issue for BoF-based representation lies in its quantization loss, which is incurred
by replacing each feature vector with its best matched
codeword(s) [11], [17], [19], [20]. To compensate, one potential solution is to improve the feature coding stage using
methods such as soft assignment [13], [17], [20], sparse
coding [18] or Hamming embedding [5]. Alternatively, a more
straightforward solution is to learn a better dictionary, such
as, but not limited to, kernelized codebook [2], hierarchical
optimization [35], and so forth. These dictionary optimization
approaches can also be combined with advanced feature coding schemes to further reduce the quantization loss.
For the purpose of image search, works in [3], [4], and [36]
proposed to hash local features into a discrete set of bins,
which are then indexed to their corresponding buckets in one
or multi hash tables. In this case, online search can be confined
within only a few buckets, leading to a sub-linear search time
with provable accuracy guarantees. In this direction, methods
such as Locality Sensitive Hashing (LSH) [3], Kernalized
LSH [4] and Spectral Hashing [36] have been widely investigated.
B. Supervised Dictionary Learning
The supervised dictionary learning paradigm [8], [23], [24],
[27], [37] exploits image/region labels or match/non-match
feature pairs to supervise the codeword generation process.
The targeted dictionary is supposed to produce semanticpreserving BoF histograms, e.g., the images belonging to
the same category should have similar BoFs and vice versa.
In detail, Mairal et al. [24] learned a dictionary in a supervised
mode by discriminative sparse coding. Lazebnik et al. [8]
learned a dictionary for the purpose of object categorization
C. Generative Dictionary Learning
In
line
with
generative
dictionary
learning,
Jaakkola et al. [43] exploited the presence/absence of class
labels to supervise the generation of BoFs using Maximum
a Posteriori. Perronnin et al. [12] used the Gaussian Mixture
Model with Fisher kernels for image categorization. In our
previous work [27], a semantic embedding scheme was
proposed to guide supervised dictionary learning, which used
a density-diversity purification to filter out imprecise patch
labels fetched from meta-data of web images. Then, a Hidden
Markov Random Field is used to leverage the purified labels
for dictionary learning.
D. Weakly Supervised Learning
Our work is inspired by an emerging machine learning
direction named “weak supervised learning” [44], [45], which
conducts a joint learning from multiple sources of labels,
in which the initial labels are not fully trusted. In practice,
a variety of real-world problems can be formalized to this
multi-labelers problem, where the reliabilities of different
annotators may vary significantly. For example, there has
been an increasing amount of endeavors exploring Amazon’s
Mechanical Turk for annotation [44]. In absence of groundtruth labels, how to simultaneously learn classifiers and
evaluate label correctness was investigated in [44] and [45],
in which a robust learning procedure is devised to deal with
multiple unreliable sources of labels.
Comparing to above, we innovate in the following aspects:
• The dictionary learning is modeled from a novel probabilistic aspect by using a generative Hidden Markov
Random Field. It enables us to explicitly model the supervised dictionary learning procedure with an ExpectationMaximization formulation.
• By exploiting the cheaply available semantic attributes
extracted from the image, the dictionary learning is
?supervised? without any explicit labels. In such a way,
our approach is label-free yet still preserves the semantic
meanings into the BoF histogram.
III. W EAKLY S UPERVISED D ICTIONARY
L EARNING F RAMEWORK
A. Generative Dictionary Learning Formulation
This section introduces the proposed Hidden Markov
Random Field (HMRF) for the formulation of generative
GAO et al.: WEAKLY SUPERVISED VISUAL DICTIONARY LEARNING
5403
Fig. 3.
Fig. 2. The graphical representation of the proposed weakly supervised
dictionary learning with attributes.
dictionary learning. In such a case, the supervised labels may
come from multiple sources, for example, image annotations,
category labels, image query logs, as well as automatic image
attribute detections (as adopted in this paper). To use supervised labels for dictionary learning, we need to link them,
directly or indirectly, to local features or codewords. For
example, labels may identify the local feature correspondence
(matching) or identify the category identity. In the latter case,
such labels are subsequently propagated to local features [27]
or codewords.
In our case, we assume that these labels are automatically
detected from images, and then propagated to codewords.
Fig. 2 shows the graphical representation of our supervised
dictionary learning formulation. This formulation consists of
three components as below:
(1) A Hidden Layer to model the label supervision,
K , each of
which consists of K nodes A = {ai }i=1
which corresponds to a label. As discussed before,
such labels may come from supervised category annotations, or alternatively from unsupervised image attribute
scores [28]–[31]. In the unsupervised case, each ai
denotes a unique attribute, which is mapped to codeM
words V = {v i }i=1
in the Observed Layer with M
mapping probabilities scored among [0, 1]. Here, K is
the dimension of semantic attributes detected, and M is
the size of the codebook learned, a.k.a., the number
of codewords (clustering centers) after quantization.
We denote A(i ) as the accumulated responses of i -th
attribute from all images in the dataset and Ak (i ) as
the response of k-th attribute from the i -th image in the
dataset.
(2) The
Mapping
represents
the
probabilistic
co-occurrences from attributes in the Hidden Layer
to codewords in the Observed Layer, as shown in
Fig. 3. We name it “Attribute-to-Codeword Mapping”
and denote it by a multi-to-multi mapping matrix
M ∈ R K ×M . The entry at the k-th row (attribute) and
the m-th column (codeword) of M is calculated as:
Mkm =
N
1 Ak (i ) × Frem (i ),
Z
(1)
The proposed attribute-to-codeword mapping.
on the i-th image. Frem (i ) is the frequency of the m-th
codeword on the i-th image.
We multiply Ak (i ) and Fm (i ) to derive the mapping
probability between the k-th attribute and the m-th
codeword for the i-th image. For each ak and v m pair,
probabilities from all reference images are summarized
to derive the final mapping score, which is then normalized among M.
After each round of dictionary learning, this mapping
is updated to provide a renewed supervision for both
the dictionary learning and the feature coding steps, as
detailed later in Section III-C.
M
(3) An Observed Layer contains codewords V = {v i }i=1
3
updated after each iteration of dictionary learning . V is
quantized from the local features extracted
from image
N
Ni N
dataset F(i ) i=1 =
f(i, l) l=1
, where f(i, l)
i=1
denotes the l-th local feature (in total Ni ) in the i-th
image. After dictionary learning, we denote the learned
N
BoF codes as U(i ) i=1 .
We take a generative understanding in the Observed
Field using the labels from the Hidden Field based on the
Attribute-to-Codeword Mapping M ∈ R K ×M . That is,
for the i-th image we treat each BoF U(i ) to be generated
from its local feature set F(i ). And this generation
process is supervised by the attribute detection scores
A(i ) of this image, formulated as:
N
P U(i )|A(i ), F(i ) .
P U|A, F =
(2)
i=1
We assume a conditional independence among individual images. Therefore, the BoF coding is solved in an
image-wise style, as detailed later in Section III-C. From
the graphical representation of Fig. 2, the BoF codes U
and attribute supervision A are conditionally dependent
given the Observed Field V. In such a manner, A is
imposed to optimize the learning procedure of BoF
codes U.
Indeed the attribute scores are determined by the
detector response rather than the inference. However,
the generation of observed layers can be still treated as
supervised (influenced) by the nodes in the hidden layer.
Here, the key replacement is not to infer the hidden
i=1
where Z is a normalization factor to ensure that each
entry in M has a maximal response of 1 and minimal
of 0. Ak (i ) is the response of the k-th attribute detector
3 Here, we do not restrict our model to the specific dictionary learning
scheme. Indeed methods like K-Means clustering [14], Hierarchical K-Means
clustering [11], or Projective Gradient Descendent [1], [18] can be fluently
adopted into our model.
5404
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 12, DECEMBER 2014
variables, but to infer the ?mapping? at each iteration,
from the learned codewords to the attributes. This mapping can then supervise the codebook quantization in
the next round.
B. Classemes Based Attribute Detection
1) Classemes Attribute Detector: In this paper, we
use the Classemes attribute descriptor proposed by
Torresani et al. in [31]. The Classemes attribute descriptor
consists of the outputs of 2,659 offline-trained object
category classifiers on a given image. The training
is done separately in a different dataset, and the
categories are selected from the Columbia LSCOM visual
concept ontology (Lexicon Definitions and Annotations,
http://www.ee.columbia.edu/ln/dvmm/lscom/). LSCOM is a
professional-designed visual concept ontology describing an
expanded multimedia concept lexicon. The LSCOM concepts
are related to events, objects, locations, people, and programs,
which are selected following a multi-step process involving
input solicitation, expert critiquing, comparison with related
ontologies, and performance evaluation.
2) Attribute Training Procedure: Classemes use the
LSCOM CYC ontology released on 2006-06-30, which contains 2,832 unique categories. Among them 97 categories
denoting abstract groups of other categories (marked in angle
brackets) are removed. Some other category removed include
plural categories that also occurred as singulars, as well as
some people-related categories which were effectively nearduplicates. Finally, they have 2,659 categories, for each of
which a corresponding detector is trained.
For the training, annotated images of a target category are
crawled from image search engines like Google Image with
hybrid features including color, texture, and local features.
Then, one-vs.-all linear SVMs are trained. Online, given a
test image, each category detector would output a probabilistic
response. The responses of all 2,659 individual detectors are
then gathered together to produce the Classemes descriptor.
3) On the Detector Accuracy and Bag-of-Words Performance: As pointed out in [31], it is not required or expected
that the base categories of Classemes will provide precise
labels, of the form water, sky, grass, beak. On the contrary,
there is an assumption that modern category recognizers are
essentially quite dumb; so a swimmer recognizer looks mainly
for water texture, and the bomberplane recognizer contains
some tuning for shapes corresponding to the airplane nose,
and perhaps the shapes at the wing and tail [31]. Even if
these recognizers are perhaps overspecialized for recognition
of their nominal category, they can still provide useful building
blocks to the learning algorithm that learns to recognize the
novel class duck.
In such a case, indeed we do not care about the explicit
performance of each individual attribute detector. Instead,
we exploit their joint response to a certain kind of images,
i.e., images with similar semantics should have similar
attribute responses, which has been clearly demonstrated
in [31] and its subsequent works. In our paper, we assume
that these detector responses, although kind of imprecise,
can provide a reasonable guess about the potential semantic
labels. This is the reason we term our approach as weakly
supervised. And the experiments have shown that such imprecise, automatic labels do significantly improve the recognition
and search accuracies. It is worth to note that, other attribute
detectors [28]–[30], [46] can be also used. And indeed, our
contribution is kind of orthogonal to the attribute detectors
chosen.
C. Supervised Dictionary Learning With Attributes
Learning both the dictionary V and the BoF codes U simultaneously is challenging, since the overall objective function
is not convex. As demonstrated in [18] and [26], the feature
coding step and the dictionary learning step are bi-convex by
fixing each other. Following this principle, we learn V and U
alternatively as below:
1) Descriptor Coding: The Markov Random Field imposes
Markov probabilistic
N distribution to supervise the learning
of V from F(i ) i=1 . In principle, given the attribute score
vector of the i-th image, the quantization of each local feature
only depends on other local features from the same image.
More specifically, for the i-th image, its BoF code Ui of
local features F(i ) only depends on F(i ) and A(i ). And the
dependency of the latter of Ui to A(i ) is measured via M:
arg min P F(i ) → U(i )|A(i )
U(i)
= arg min
U(i)
+
Ni
||V Q f(i, l) − f(i, l)||2
l=1
Ni K
l=1 k=1
α × exp − M Q f(i,l),k × Ak (i ) ,
(3)
where F(i ) → U(i ) denotes the quantization operation for
local features F(i ) extracted from the i-th image, which
is
specified by quantizing each local feature f(i, l) as Q f(i, l) .
V(k) is the k-th codeword. The objective function in Equation 3 is to learn the bag-of-words code U (i ) for the i -th
image, during which procedure the quantizer (codebook) V is
fixed.
Q f(i, l) is the quantizer that encodes a local feature f(i, l)
into one or more codewords in V, which outputs
the quantized
codeword(s). The definition of quantizer Q f(i, l) is very
flexible, for instance hard quantization [11], [14], [27], soft
quantization [13], [17], and sparse coding [18], [26].
Looking into the attribute supervision term in Equation (3),
we add a violation penalty term M Q f(i,l),k ×Ak (i ) with expo
nential loss into the descriptor coding loss ||V Q f(i, l) −
f(i, l)||2 . To obtain the attribute scores A(i ) for each image, we
adopt the Classemes attribute detector [31], which produces a
2,659-dimensional attribute vector for a given image. The time
cost is typically less than 0.5 second for an image of 400×600
resolution on a regular PC.4
4 Without loss of generality, other visual attribute detectors, such
as [28]–[30], [32] can be also used here. This choice is orthogonal to the
core contribution of this paper.
GAO et al.: WEAKLY SUPERVISED VISUAL DICTIONARY LEARNING
5405
a) Explanation: It is worth to note that, a straightforward
alternative solution is to re-weight the BoF matching scores
using the attribute scores. This solution can be treated as
a “late fusion”, which carries out the combination of BoF
based and attribute based scores at the stage of similarity
ranking. In contrast, our approach can be regarded as an
“early fusion”, which combines the BoF code and attribute
scores at the feature extraction stage. In Section IV, we will
quantitatively show that the proposed approach works better
than this alternative solution.
The key of our approach is to introduce the attributebased automatic semantic annotation to “supervise” the coding
process as in Equation 3. This is reflected by adding the
semantic-based loss term as in the last line of Equation 3.
To do this, two operations are designed, i.e., 1. attribute detection for individual images and 2. learning the overall attributecodeword mapping function M. In such a manner, we can
embed semantics into the resulting BoF. Subsequently, images
with related attributes are supposed to have similar BoF histograms. This is beneficial in many applications. For instance,
if this BoF histogram is used for image search, not only
the visually similar images, but also the semantically related
images to the query image, would be returned. For another
instance, if this BoF histogram is used for image categorization, images belonging to identical semantic category would
be more likely to be grouped together in the feature space.
Therefore, it is easier to train an accurate classifier, even using
a linear model (e.g., linear SVM as we tested in this paper).
2) Dictionary Learning: In this step, we aim to minimize
the local feature quantization error. This is achieved by
M
from
re-estimating its M codeword centroid vectors {v m }m=1
the local feature assignments done in the previous coding step.
For the k-th codeword, we update its centroid vector vk by
using local features that are assigned into this codeword:
fi ∈vk fi
vk =
(4)
s.t. vk = fi |Q(fi ) = vk .
|vk |
Note that the above Q(fi ) = vk describes the case of
one-to-one assignment from feature fi to codeword vk . And it
is straightforward to extend to multi-to-one case.
Algorithm 1 describes the iterative procedure between the
local feature coding and the dictionary learning steps. For
simplicity, from now on we rewrite the local features and their
N
N
and C = {ci }i=1
, which are fully equal
indices as F = {fi }i=1
N
N
to F(i ) i=1 and {Ci }i=1 , respectively. And N is the number
of local features extracted from the entire image dataset.
D. Correlation to Existing Dictionary Learnings
In a simplified case, a hard quantization [11], [14] assigns
the i-th local feature fi into only one codeword, such that the
following quantization loss is minimized:
min
V,C
M
||f j − vi ||2 = min
k=1 Q(f j )=vi
V,C
N
||fi − Vci ||2
where the second to the fourth lines in Equation (5) constrain
that only one word is assigned to the local descriptor fi .
To reduce the one-to-one quantization error, this hard constraint is usually relaxed into a soft assignment [17], [20],
e.g., removing the |ci | = 1 constraint in Equation (5).
To further avoid over fitting, i.e., assigning each local feature
to too many codewords, recent works [18], [26] have proposed
to impose an 1 regularizer into Equation (5), which results
in a sparse coding based objective function:
N fi − Vci 2 + λci 1
min
V,C
i=1
s.t. ∀i ∀ j ci j ∈ [0, 1], ci, j ≥ 0.
(6)
Without loss of generality, both Equation (5) and Equation (6)
can be directly integrated into our proposed dictionary learning
formulation in Equation (3). Taking the sparse coding for
instance, the following representation can be easily derived:
min
V,C
N ||fi − Vci ||2 + λ||ci ||1
i=1
+
Ni K
l=1 k=1
α × exp − M Q f(i,l),k × Ak (i )
s.t. ∀i ∀ j ci j ∈ [0, 1], ci, j ≥ 0.
(7)
IV. E XPERIMENTAL E VALUATIONS
We carried out two groups of experimental evaluations.
First, in Section IV-C, we evaluate the proposed approach in
the task of image classification, with comparisons to the stateof-the-art works [18], [33], [34] on Scene15, Caltech256, and
ILSVRC2010 datasets. Second, in Section IV-D, we evaluate
the proposed approach in the task of image search, with
comparisons to the state-of-the-art works [5], [18], [24], [25]
on Holiday and NUS-WIDE benchmark, as well as a
one-million landmark image dataset.
i=1
s.t. ∀i |ci | = 1,
∀ j ci j = {0, 1},
∀i ∀ j ci, j ≥ 0,
Algorithm 1 The Bi-Convex Optimization Procedure of
the Proposed Weakly Supervised Dictionary Learning With
Attributes
A. Baselines and Evaluation Protocols
(5)
In the task of image classification, we compare our dictionary learning approach to ScSPM [18] and LScSPM [25], both
5406
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 12, DECEMBER 2014
of which are very related to the proposed approach. Given the
fact that the spatial cues are widely recognized as being helpful
for BoF representation, we further combine our approach and
the alternative approaches of [18] and [25] with the scheme
proposed in [34], which adopts average pooling + Spatial
Pyramid Matching (SPM) to integrate spatial cues. We compare our approach combined with [34] to two alternative
solutions, i.e., (1) Weak Geometric Consistency + Hamming
Embedding proposed in [5] and (2) Max Pooling with ScSPM
proposed in [18]. For SPM, we use a 3-layer, 4-division setting
with identical weights for each division in each layer. For
ScSPM [18], we directly run their codes for Caltech256 with
the default settings. On both Scene15 and Caltech256, we use
the classification precision per class as the evaluation protocol,
which is identical to [18], [25], [34] and allows us to directly
compare to the results reported in [18], [25], and [34].
In the task of image search, we use mean Average Precision (mAP) as the evaluation protocol, which is widely used
in related work such as [6], [7], and [14]. mAP is defined as
follows:
l
L Nrelevant
P(r )
1 r=1
,
(8)
m AP =
l
L
Nrelevant
l=1
l
where l = 1 to L is the total queries for evaluation. Nrelevant
is the number of relevant images to the l-th query; r is the the
r-th relevant image; P(r ) is the precision at the cut-off rank
of image r .
The goal of evaluation on the image search experiment is
to validate that the proposed supervised dictionary learning
approach can generate BoF histograms that are semanticpreserving. That is, if two images have similar or related
semantic meaning, i.e. both containing similar scene layouts
and composition, their corresponding BoF histograms generated using our approach should be more similar, comparing
to the original BoF histograms generated without semantic
supervision. In this way, given the query image, it is more
likely to find semantically similar images from the database. From another perspective, this benefit can be regarded
as a solution to handle the so-call semantic gap, i.e. the
gap between visual representation and high-level semantic
meaning, which is a long-standing research challenge in
content-based image retrieval.
It is worth to note that, adding supervised information
to facilitate image retrieval is an interesting and ongoing
research, with several directions such as visual reranking,
pseudo relevance feedback, and query-dependent hashing,
etc. Our work differs from the existing ones and merits in
directly embedding the supervised information into the feature
representation. In such a manner, the subsequent procedures
like inverted indexing, distance calculation, can be retained.
Therefore, we do not need to change or add new components
into the standard search or indexing structure, which is very
helpful for large-scale applications.
B. Parameter Tunings and Implementation Details
1) Feature Sampling: In the stage of local feature extraction, to make a fair comparison, we follow the setting of [18]
TABLE I
C LASSIFICATION P RECISIONS ON S CENE 15
to do dense sampling in a given image. In such a case, we
first resize the max of length or width to 300 pixels with the
same aspect ratio. Then, we densely sample local patches in
the resized image on a regular grid, with the step and patch
sizes as 8 and 16, respectively. Each sampled local patch is
described using SIFT [10], which results in a 128-dim feature
vector.
2) Dictionary Size: Nevertheless, an optimal performance
can be achieved by tuning the dictionary size via crossvalidation. However, we prefer to fix this size to compare
among different dictionary learning approaches. Under this circumstance, we fix the dictionary size for image classification
as 1,024, as learned from (1.0 ∼ 1.2) × 105 randomly selected
local features from each dataset respectively. For test on the
ILSVRC2010 dataset, we further train a 8, 000-dim dictionary
as to be comparable to the codebook size used in [48].
3) Sparsity Regularization: In the stages of both sparse
coding and max/average pooling, we use the codes released
in [18]. The sparsity of the sparse codes λ is set as 0.3-0.4,
which is reported to achieve the best performance in [18].
4) Classifier: In the stage of classifier training, following [18], [25], we adopt a one-vs.-all Linear SVM setting,
which is shown to be quite effective and efficient by combining
with max pooling, as demonstrated in [18] and [25]. Note that,
without loss of generality, other classifiers can be also used,
which is orthogonal to the contribution of this paper.
C. Image Classification Performance
In the image classification task, we evaluate on the Scene15,
the Caltech256, and the ILSVRC2010 datasets.
1) Scene15: Scene 15 contains 15 categories with
4,485 images in total, where each category contains about
200 to 400 images (http://www-cvr.ai.uiuc.edu/ponce_grp/
data/scene_categories/scene_categories.zip). The image content is diverse, ranging from indoor to outdoor scenes, for
instance bedroom, kitchen building and country, etc. In comparison, we randomly select 100 images per class for training,
and then use all rest images for testing.
Table I lists the performances of our approach and alternative methods. Fig. 4 shows the confusion matrix for scenes. It
is clear that our weakly supervised dictionary learning scheme
has largely outperformed the state-of-the-art approaches. Also,
it outperforms our alternative approach, which trains Linear
SVMs on BoF and Classemes vectors respectively, and then
fuses their regression scores. This result indicates that, for the
purpose of using attributes and BoF features jointly, our “early
fusion”-like combination is more suitable than the above “late
fusion”-like combination.
GAO et al.: WEAKLY SUPERVISED VISUAL DICTIONARY LEARNING
5407
TABLE III
F LAT C OST T EST ON 5 P REDICTIONS ON I MAGE N ET 2010 U SING O UR
S EMANTIC -P RESERVING BoF + S TOCHASTIC SVM, W ITH C OMPARISONS
TO
Fig. 4. Confusion matrix of the proposed weakly supervised dictionary
learning scheme on Scene15 (vertical: original label, horizontal: inferred
label).
TABLE II
P ERFORMANCE C OMPARISONS ON C ALTECH 256
2) Caltech256: Caltech256 contains 256 categories and
29,780 images with a background class (www.vision.
caltech.edu/Imagedatasets/Caltech256/). We evaluate our
method under four different settings: i.e. 15, 30, 45 and
60 training images per category respectively. The results on
this dataset are listed in Table II. From Table II, we can
observe that our method has outperformed these state-ofthe-art approaches. Moreover, our method is more robust,
as indicated by the lowest variances in accuracy. Finally,
our approach is fully unsupervised in the stage of feature
extraction, i.e., without requiring any category labels. This
serves as an important advantage comparing to the alternative
approaches of [18] and [25] that perform most closely to our
approach.
In addition, we also compare our approach in combination with Radius Basis Function to the Fisher Kernel
based approach proposed in [47], which can be treated as
a non-linear feature mapping approach with linear SVM.
Similarly, the proposed feature is also a sort of non-linear
feature mapping from the image. By combining with linear
SVM, the proposed feature performs worse than Fisher Kernel based approach. However, by combing with non-linear
SVM, the proposed feature can be treated as the non-linear
augmented in the feature end. As shown in the second table of
Table II, in such a way our approach can still outperform this
state-of-the-art alternative approach.
It is also possible to include performance of other supervised dictionary learning approach and compare in Table II.
However, LScSPM is already the state-of-the-art work in the
literature. Therefore, by outperforming LScSPM, the merit of
our approach has already been demonstrated.
S TATE - OF - THE -A RTS OF LCC [48] AND F ISHER K ERNEL [47],
AS W ELL AS B ASELINE B O F M ETHODS
3) ILSVRC2010: Furthermore, we have conducted a
group of experiments to verify the performance of our
semantic-preserving BoF in the scenario of large-scale object
recognition. To this end, we test both our feature and
several alternative and state-of-the-art approaches on the
ILSVRC2010 dataset (http://www.image-net.org/challenges/
LSVRC/2010/download-public). This dataset was used for
ImageNet 2010 challenge, which contains 1.2M images from
1,000 diverse categories. There are a total of 1,261,406 images
for training. The number of images for each synset (category)
ranges from 668 to 3047. There are 50,000 validation images,
with 50 images per synset.
We extracted several groups of features in this experiment.
First, we use the dense SIFT features provided on the website,
which are extracted using VLFeat based implementation with
10 pixels spacing and 20 × 20 overlapping patches. We further
extract HoG + LBP features following the setting of [48],
with three scales of patch size for computing HOG + LBP,
namely, 1616, 2424 and 3232. We then concatenate both HoG
and LBP features into a single feature vector. On that, we run
a codebook with dimension 8,000, which approximates the
codebook size of [48]. We use stochastic SVM (i.e., stochastic
gradient descendent SVM) as classifier.5
We then compare our the learned semantic-preserving BoF
feature with: (1) Descriptor Coding (HOG + LBP) + stochastic SVM [48], which is reported as the best performed
method in ImageNet 2010, (2) Fisher Kernel + SVM [47],
which is the second best in the ImageNet 2010 challenge,
(3) Semantic-preserving BoF (Dense Sift) + stochastic SVM,
and finally (4) BoF (Dense Sift) + stochastic SVM. Tab. III
shows the performance comparison between our approach
(a.k.a., Semantic-preserving BoF (HOG + LBP) + stochastic
SVM) and the above four alternative approaches. To directly
compare to the results reported in the ImageNet 2010 challenge, we use Flat Cost with 5 Prediction as our evaluation
protocol.
As shown in Tab.III, it is quite clear that the proposed semantic-preserving BoF outperforms LCC based
approach [48] with the same feature setting (HOG + LBP).
Also, by only using SIFT as local features, our semanticpreserving BoF also significantly outperforms the baseline
BoF with a large margin (Flat Cost 0.47 to 0.78). It is worth to
note that, in our case the sparse coding step is very efficient.
5 We use three servers each with 16-core CPU to train these 1000 categories,
which costs approximately 4 days for training. The extraction of HoG feature
is also paralleled using these 48 cores, which costs approximately one week
for extracting both HoG and LBP features.
5408
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 12, DECEMBER 2014
Fig. 5. Performance comparisons on Holidays + Flickr1M, with codebook
size = 20, 000.
TABLE IV
mAP C OMPARISONS TO V OCABULARY T REE [11], SPM [34]
AN AND
Fig. 6. Performance comparisons on Holidays + Flickr1M, with codebook
size = 200, 000
ScSPM [18] ON NUS-WIDE
Also, the performance of the proposed semantic-preserving
BoF can be further improved by using non-linear SVMs,
e.g., by using the chi2 kernel.
D. Image Search Performance
In the task of image search, we evaluate on both the
Holidays + Flickr 1M and the NUS-WIDE benchmarks, as
well as a 1-million landmark image dataset.
1) Holidays + Flickr1M: First, we validate our performance
on the Holidays + Flickr1M dataset [5], which is constructed
by combining the Holidays near-duplicate image search benchmark and one-million Flickr distractor images. On such a large
dataset, the outliers of labels are unavoidable, which seriously
degenerate the performance of other supervised dictionary
learning approaches [18], [24], [25]. In contrast, by exploring
only the attribute-level weak supervision, our approach does
not require any actual labels. In the following setting, we do
not compare approaches using supervised dictionary learning
in the image search task.
As shown in Fig. 5 and Fig. 6, our approach also
significantly outperforms unsupervised dictionary learning
approaches, including Vocabulary Tree [11], SPM [34], Weak
Geometric Consistency and Hamming Embedding [5]. It is
worth to note that the Holiday benchmark is designed for
the application of landmark retrieval, while the Classemes
attribute used in our approach is designed for general-purpose
image description. Even given such an application bias, the
combination of attributes to dictionary learning still achieves
better performance, which is due to the generality of attributebased detectors. This again proves the correctness and the
generality of the proposed approach.
Fig. 7. Performance comparison of the proposed approach to alternatives
[11], [34], [35] in the ten-million landmark image dataset.
2) NUS-WIDE: The second dataset is NUS-WIDE, which
contains around 270,000 web images associated with
81 ground truth concept tags (http://lms.comp.nus.edu.
sg/research/NUS-WIDE.htm). Each image in NUS-WIDE
contains multiple tags, which serves as the ground truth
categories to be used for supervised dictionary learning.
In evaluation, we consider 21 most frequent tags, like “animal”, “buildings”, “person”, etc., each of which has abundant
relevant images ranging from 5,000 to 30,000. We sample
uniformly 100 images from each of the selected 21 tags to
form a query set of 2,100 images, while the rest images are
left to build the training set. Using the learned dictionary, each
image is represented by a 1,024-dimensional feature vector,
similar to LLC [26].
Table IV shows that our approach (weakly unsupervised
dictionary learning + max or average pooling) significantly
outperforms baselines including Vocabulary Tree [11],
SPM [34] and ScSPM [18]. As for SPM [34] and ScSPM [18],
our performance gain can be explained by the fact that only
a small portion of images in the NUS-WIDE dataset is
manually labeled. On the contrary, lots of other images in the
NUS-WIDE dataset are based on their original meta tagging,
without any correctness checking. Indeed, most of such tags
GAO et al.: WEAKLY SUPERVISED VISUAL DICTIONARY LEARNING
5409
Fig. 8. Examples in our 10 Million landmark image dataset, in which images are crawled from Flickr and Panoramia from cities of Beijing, New York City,
Barcelona, Singapore and Florence.
Fig. 9. Confusion table of our approach (bottom) on PASCAL VOC dataset
comparing to [15] (top).
are noisy and degenerate the supervised dictionary learning for
ScSPM [18]. However, such noise does not have influence on
our approach, since our approach does not require any actual
labels.
3) 1M Landmarks: We also test the performance of our
approach for image search using a 1-million landmark image
data set collected from Flickr (www.flickr.com) and Panoramia
(www.Panoramia.com). We crawl images with their GPS tags
from 5 metropolitans worldwide, including Beijing, New York
City, Barcelona, Singapore and Florence, as shown in Fig. 8.
This dataset is named as 1M Landmarks in our subsequent
discussion.
We run k-means clustering by using the GPS tags to
partition photos of each city into multiple regions. For each
city, we select the top 30 densest regions as well as 30 random
regions. We then ask a group of volunteers to identify one or
more dominant landmark views from each of these 60 regions.
For a given dominant view, all their near-duplicate photos are
manually labeled from the regions it locates and surroundings
regions nearby. Eventually we have 300 queries as well as
their ground truth labels (corrected matches).
Since it is not suitable to define such landmark-level identity
as supervised information, here we only compare our solution
to unsupervised dictionary learning. Fig. 7 presents the mAP
comparison between our approach and three unsupervised dictionary learning approaches, including Vocabulary Tree [11],
SPM [34] and Hierarchical Vocabulary Optimization [35].
To clarify the contributions of different components in our
approach, we derive two alternatives in terms of (1) Max or
Average Pooling + Spatial Pyramid and (2) Sparse Coding.
The performance of above 5 baselines are shown in Fig. 7. It is
clear that our dictionary learning approach significantly outperforms the other three baselines [11], [34], [35]. We have also
discovered that, the baseline SPM indeed does not introduce
significant performance gain, while in some settings of the
dictionary size, the corresponding mAP even degenerates. This
phenomenon can be explained by the fact that, in an uncontrolled setting, we cannot guarantee a uniform distance to take
photos of landmarks. Therefore, using SPM is inappropriate,
as the spatial layouts of corresponding photos might vary a
lot. Furthermore, max pooling is shown to perform better than
average pooling. We explain this phenomenon by the ability
of max pooling to capture the salient response in each spatial
subdivision. In contrast, average pooling only simply records
the averaged responses in each spatial subdivision.
4) Comparison to Supervised Dictionary Learning: We further compare our approach to two supervised dictionary learning approaches [15], [27] on PASCAL VOC 2005 (http://www.
PASCAL-network.org/challenges/VOC/). We adopt VOC 05
not 08 here, as to directly compare our results to that
of [15] and [27], which are most related to our approach.
We split PASCAL into two equal-sized sets for training and
testing, as identical to [15]. The training set provides bounding
boxes for image regions with annotations, which gives the
purified patch-label correspondences for implementing [27].
As for [27], the correspondence set of local features is built
from the bounding boxes with annotation labels, from which
the supervised dictionary is learned. In classifier learning, each
annotation is viewed as a class with a set of BoFs extracted
from the corresponding bounding boxes. Identical to [15], the
classification of the test bounding box is accomplished by
using nearest neighbor search.
As shown in Figure 9, in almost all categories, our method
achieves better precision comparing to [15] and [27]. Note that,
the supervised dictionary learning of [27] actually provides
almost comparable performance to our dictionary learning
scheme. However, our solution does not depend on any actual
labels. This advantage is important especially to cope with the
increase of category numbers. For instance in the ImageNet
5410
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 12, DECEMBER 2014
evaluations, the category number could be tens of thousands
(or even more). In such a case, the label correlations would be
a serious issue, which would largely challenge the performance
of the existing supervised dictionary learning schemes.
V. C ONCLUSIONS AND F UTURE W ORK
While most existing dictionary learning methods resort to
unsupervised quantization, the recent trend prefers supervised
dictionary learning by incorporating manual annotations or
category labels of images. However, the expensive labor of
human labeling hampers supervised dictionary learning from
real-world, large-scale computer vision tasks. To the best
of our knowledge, this paper is the first study on how to
make use of cheaply available image attributes to help the
visual dictionary learning process, accomplishing in a weakly
supervised learning formulation. We tackle dictionary learning
from a generative learning perspective. We propose a Hidden
Markov Random Field (HMRF) model to effectively and
robustly exploit the error-containing supervision stemming
from the image attributes. In the HMRF model, the attributes
are modeled into a hidden layer, while the codewords are
modeled into an observation layer. As such, the codewords
are iteratively refined through updating an attribute-to-word
mapping.
The proposed weakly supervised dictionary learning
approach is generic, because some involved components (e.g.,
feature coding and attribute detection) in our approach can be
easily replaced by other cutting-edge algorithms. Meanwhile,
our formulation can cover several existing unsupervised and
supervised dictionary learning methods. In evaluation, we
demonstrate the remarkable advantages of our approach in
terms of image classification and search experiments performed on several benchmark datasets.
R EFERENCES
[1] J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra, “Efficient
projections onto the 1 -ball for learning in high dimensions,” in Proc.
25th Int. Conf. Mach. Learn., 2008, pp. 272–279.
[2] J. C. van Gemert, C. J. Veenman, A. W. M. Smeulders, and
J.-M. Geusebroek, “Visual word ambiguity,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 32, no. 7, pp. 1271–1283, Jul. 2010.
[3] A. Gionis, I. Piotr, and M. Rajeev, “Similarity search in high dimensions
via hashing,” in Proc. Int. Conf. Very Large Data Bases, 1999,
pp. 518–529.
[4] B. Kulis and K. Grauman, “Kernelized Locality-Sensitive Hashing for
Scalable Image Search,” in Proc. IEEE Int. Conf. Comput. Vis., 1999,
pp. 2130–2137.
[5] H. Jegou, M. Douze, and C. Schmid, “Hamming embedding and weak
geometric consistency for large scale image search,” in Proc. 10th Eur.
Conf. Comput. Vis., 2008, pp. 304–317.
[6] H. Jegou, M. Douze, C. Schmid, and P. Perez, “Aggregating local
descriptors into a compact image representation,” in Proc. IEEE Int.
Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 3304–3311.
[7] R. Ji et al., “Location discriminative vocabulary coding for mobile landmark search,” Int. J. Comput. Vis., vol. 96, no. 3, pp. 290–314, Jul. 2011.
[8] S. Lazebnik and M. Raginsky, “Supervised learning of quantizer
codebooks by information loss minimization,” IEEE Trans. Pattern
Anal. Mach. Intell., vol. 31, no. 7, pp. 1294–1309, Jul. 2009.
[9] J. Liu, Y. Yang, and M. Shah, “Learning semantic visual vocabularies
using diffusion distance,” in Proc. IEEE Int. Conf. Comput. Vis. Pattern
Recognit., Jun. 2009, pp. 461–468.
[10] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”
Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.
[11] D. Nister and H. Stewenius, “Scalable recognition with a vocabulary
tree,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.,
vol. 2. Jun. 2006, pp. 2161–2168.
[12] F. Perronnin, C. Dance, G. Csurka, and M. Bressan, “Adapted
vocabularies for generic visual categorization,” in Proc. 9th Eur. Conf.
Comput. Vis., May 2006, pp. 464–475.
[13] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman, “Object
retrieval with large vocabularies and fast spatial matching,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2007, pp. 1–8.
[14] J. Sivic and A. Zisserman, “Video Google: A text retrieval approach to
object matching in videos,” in Proc. 9th IEEE Int. Conf. Comput. Vis.,
Oct. 2003, pp. 1470–1477.
[15] J. Winn, A. Criminisi, and T. Minka, “Object categorization by learned
universal visual dictionary,” in Proc. IEEE Int. Conf. Comput. Vis.,
vol. 2. Oct. 2005, pp. 1800–1807.
[16] Z. Wu, Q. Ke, M. Isard, and J. Sun, “Bundling features for large scale
partial-duplicate web image search,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., Jun. 2009, pp. 25–32.
[17] J. Yang, Y.-G. Jiang, A. G. Hauptmann, and C.-W. Ngo, “Evaluating
bag-of-visual-words representations in scene classification,” in Proc.
Int. Workshop Workshop Multimedia Inf. Retr., Sep. 2007, pp. 197–206.
[18] J. Yang, K. Yu, Y. Gong, and T. Huang, “Linear spatial pyramid
matching using sparse coding for image classification,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 1794–1801.
[19] O. Boiman, E. Shechtman, and M. Irani, “In defense of nearest-neighbor
based image classification,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., Jun. 2008, pp. 1–8.
[20] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman, “Lost
in quantization: Improving particular object retrieval in large scale
image databases,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.,
Jun. 2008, pp. 1–8.
[21] Y. LeCun et al., “Handwritten digit recognition with a back-propagation
network,” in Advances in Neural Information Processing Systems.
San Francisco, CA, USA: Morgan Kaufmann, Nov. 1989.
[22] T. Serre, L. Wolf, and T. Poggio, “Object recognition with features
inspired by visual cortex,” in Proc. IEEE Comput. Soc. Conf. Comput.
Vis. Pattern Recognit., Jun. 2005, pp. 994–1000.
[23] F. Moosmann, B. Triggs, and F. Jurie, “Fast discriminative visual
codebooks using randomized clustering forests,” in Advances in Neural
Information Processing Systems. Cambridge, MA, USA: MIT Press,
Dec. 2007, pp. 985–992.
[24] J. Mairal, F. R. Bach, J. Ponce, G. Sapiro, and A. Zisserman, “Supervised
dictionary learning,” in Advances in Neural Information Processing
Systems. Red Hook, NY, USA: Curran & Associates Inc., Dec. 2008.
[25] S. Gao, I. W. Tsang, L.-T. Chia, and P. Zhao, “Local features are not
lonely—Laplacian sparse coding for image classification,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 3555–2561.
[26] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, “Localityconstrained linear coding for image classification,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., Jun. 2010, pp. 3360–3367.
[27] R. Ji, H. Yao, X. Sun, B. Zhong, and W. Gao, “Towards semantic
embedding in visual vocabulary,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., Jun. 2010, pp. 918–925.
[28] V. Ferrari and A. Zisserman, “Learning visual attributes,” in Advances
in Neural Information Processing Systems. Red Hook, NY, USA:
Curran & Associates Inc., Dec. 2007.
[29] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, “Describing objects
by their attributes,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.,
Jun. 2009, pp. 1778–1785.
[30] Y. Wang and G. Mori, “A discriminative latent model of object classes
and attributes,” in Proc. 11th Eur. Conf. Comput. Vis., Sep. 2010,
pp. 155–168.
[31] L. Torresani, M. Szummer, and A. Fitzgibbon, “Efficient object category
recognition using classemes,” in Proc. 11th Eur. Conf. Comput. Vis.,
Sep. 2010, pp. 776–789.
[32] B. Siddiquie R. S. Feris, and L. S. Davis, “Image ranking and retrieval
based on multi-attribute queries,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., Jun. 2011, pp. 801–808.
[33] Z. Wu, Q. Ke, M. Isard, and J. Sun, “Bundling features for large scale
partial-duplicate web image search,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., Jun. 2009, pp. 25–32.
[34] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features:
Spatial pyramid matching for recognizing natural scene categories,” in
Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2.
Jun. 2006, pp. 2169–2178.
[35] R. Ji, X. Xie, H. Yao, and W.-Y. Ma, “Vocabulary hierarchy optimization
for effective and transferable retrieval,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit., Jun. 2009, pp. 1161–1168.
[36] Y. Weiss, A. Torralba, and R. Fergus, “Spectral hashing,” in Advances
in Neural Information Processing Systems. Red Hook, NY, USA:
Curran & Associates Inc., 2008.
GAO et al.: WEAKLY SUPERVISED VISUAL DICTIONARY LEARNING
5411
[37] R. Ji, H. Yao, W. Liu, X. Sun, and Q. Tian, “Task-dependent visualcodebook compression,” IEEE Trans. Image Process., vol. 21, no. 4,
pp. 2282–2293, Apr. 2012.
[38] T. Kohonen, Self Organizing Maps, 3rd ed. New York, NY, USA:
Springer-Verlag, 2000.
[39] A. Rao, D. Miller, K. Rose, and A. Gersho, “A generalized VQ method
for combined compression and estimation,” in Proc. IEEE Int. Conf.
Acoust., Speech, Signal Process., vol. 4. May 1996, pp. 2032–2035.
[40] F.-F. Li and P. Pietro, “A Bayesian hierarchical model for learning
natural scene categories,” in Proc. IEEE Comput. Soc. Conf. Comput.
Vis., vol. 2. Jun. 2005, pp. 524–531.
[41] R. Ji, J. Chen, L.-Y. Duan, and W. Gao, “Towards compact topical
descriptors,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.,
Jun. 2012, pp. 2925–2932.
[42] A. Bosch, A. Zisserman, and X. Munoz, “Scene classification using a
hybrid generative/discriminative approach,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 30, no. 4, pp. 712–727, Apr. 2008.
[43] T. S. Jaakkola and D. Haussler, “Exploiting generative models
in discriminative classifiers,” in Advances in Neural Information
Processing Systems. Cambridge, MA, USA: MIT Press, 1999.
[44] V. C. Raykar et al., “Supervised learning from multiple experts: Whom
to trust when everyone lies a bit,” in Proc. 26th Annu. Int. Conf. Mach.
Learn., 2009, pp. 889–896.
[45] O. Dekel and O. Shamir, “Good learners for evil teachers,” in Proc.
26th Annu. Int. Conf. Mach. Learn., 2009, pp. 233–240.
[46] L.-J. Li, H. Su, L. Fei-Fei, and E. P. Xing, “Object bank: A highlevel image representation for scene classification & semantic feature
sparsification,” in Advances in Neural Information Processing Systems.
Red Hook, NY, USA: Curran & Associates Inc., 2010, pp. 1378–1386.
[47] F. Perronnin, J. Sánchez, and T. Mensink, “Improving the Fisher kernel
for large-scale image classification,” in Proc. 11th Eur. Conf. Comput.
Vis., 2010, pp. 143–156.
[48] Y. Lin et al., “Large-scale image classification: Fast feature extraction
and SVM training,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.,
Jun. 2010, pp. 1689–1696.
Yue Gao (SM’14) received the B.S. degree from
the Harbin Institute of Technology, Harbin, China,
and the M.E. and Ph.D. degrees from Tsinghua
University, Beijing, China.
Rongrong Ji (SM’14) received the Ph.D. degree
in computer science from the Harbin Institute of
Technology, Harbin, China. He was a Post-Doctoral
Research Fellow with Columbia University, New
York, NY, USA, from 2011 to 2013. He has authored
over 40 referred journals and conferences in the
International Journal of Computer Vision, the IEEE
T RANSACTIONS ON I MAGE P ROCESSING, the IEEE
T RANSACTIONS ON M ULTIMEDIA, the ACM Transactions on Multimedia Computing, Communications
and Applications, the IEEE M ULTIMEDIA, Pattern
Recognition, the Computer Vision and Pattern Recognition (CVPR) Conference, the ACM Multimedia Conference, the International Joint Conference
on Artificial Intelligence, and the Conference on Artificial Intelligence. His
research interests include image and video search, content understanding,
mobile visual search and recognition, and interactive human–computer interface. He was a recipient of the Best Paper Award at the ACM Multimedia
2011 and the Microsoft Fellowship 2007. He is the Guest Editor of the
IEEE Multimedia Magazine, the IEEE N EUROCOMPUTING, the IEEE Signal
Processing Magazine, the ACM Multimedia Systems, and Multimedia Tools
and Applications. He was the Special Session Chair of the International
Conference on Multimedia Retrieval 2014, the Visual Communications and
Image Processing Conference 2013, the Magnetism and Magnetic Materials
Conference 2013, and the Pacific & Rim Conference on Multimedia 2012. He
was also the Program Committee Member of over 30 flagship international
conferences, including CVPR 2013, the International Conference on Computer
Vision 2013, and the ACM Multimedia 2014 to 2010. He is currently a
Professor with the Department of Cognitive Science, School of Information
Science and Engineering, Xiamen University, Xiamen, China.
Wei Liu (M’14) received the Ph.D. degree from
Columbia University, New York, NY, USA, in 2012.
He was a recipient of the 2013 Jury Award for Best
Thesis of Columbia University. He is currently a
research staff member of the IBM Thomas J. Watson
Research Center, Yorktown Heights, NY, USA, with
research interests in machine learning, computer
vision, pattern recognition, and information retrieval.
Qionghai Dai (SM’05) received the B.S. degree
in mathematics from Shanxi Normal University,
Xi’an, China, in 1987, and the M.E. and Ph.D.
degrees in computer science and automation from
Northeastern University, Shenyang, China, in 1994
and 1996, respectively. Since 1997, he has been
with the faculty of Tsinghua University, Beijing,
China, where he is currently a Professor and the
Director of the Broadband Networks and Digital
Media Laboratory. His research areas include signal
processing, broadband networks, video processing,
and communication.
Gang Hua (SM’11) received the B.S. degree in
automatic control engineering and the M.S. degree
in control science and engineering from Xi’an
Jiaotong University (XJTU), Xi’an, China, in 1999
and 2002, respectively, and the Ph.D. degree from
the Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL,
USA, in 2006. He was enrolled in the Special Class
for the Gifted Young of XJTU in 1994. He is currently an Associate Professor of Computer Science
with the Stevens Institute of Technology, Hoboken,
NJ, USA. He was a research staff member of the IBM Research Thomas J.
Watson Center, Hawthorne, NY, USA, from 2010 to 2011, a Senior Researcher
with the Nokia Research Center, Hollywood, CA, USA, from 2009 to 2010,
and a Scientist with the Microsoft Live Labs Research, Redmond, WA, USA,
from 2006 to 2009. He has authored over 50 peer-reviewed publications
in prestigious international journals and conferences. He holds three U.S.
patents and 17 more patents pending. He is an Associate Editor of the IEEE
T RANSACTIONS ON I MAGE P ROCESSING and the IAPR Journal of Machine
Vision and Applications, and a Guest Editor of the IEEE T RANSACTIONS ON
PATTERN A NALYSIS AND M ACHINE I NTELLIGENCE and the International
Journal on Computer Vision. He was the Area Chair of the 2011 IEEE
International Conference on Computer Vision and the 2011 ACM Multimedia,
and the Workshops and Proceedings Chair of the 2011 IEEE Conference on
Face and Gesture Recognition. He is a member of the Association for the
Computing Machinery.