Finding Celebrities in Billions of
Web Images
云飞
2012-12-13
Overview
• 一、label an input image with a list of
celebrities.
• 二、the celebrity names are assigned to the
faces by label propagation on a facial
similarity graph.
Overview
• 本文的优点：
• 1、the proposed image annotation system is
capable of labeling names to general web
images.
• 2、our name assignment algorithm does not
impose any assumption on the facial feature
distribution.
• 3、not only visual cues are used.
Overview
• 1. determine, by identifying celebrity names
from surrounding text.
• 2. given a set of names, assign the names to
the faces in the input image.
Overview
• A. Image Annotation System
• 1) construct a vocabulary;
• 2) discover all webpages hosting its near-duplicates;
• 3) use the vocabulary to filter the surrounding text.
– Advances：
• 1)effective;
• 2)remove noise.
– Annotated images:
• 1)SFSN
• 2)SFMN
• 3)MF
Overview
• B. Multimodal Name Assignment
• The context likelihood incorporates the information
from surrounding text by using the confidence scores
estimated by the image annotation system.
IMAGE ANNOTATION SYSTEM
• Goal: label an input image with a list of
celebrities who may appear in the image.
• A. Constructing a Large-Scale Celebrity Name
Vocabulary
• B. Discover Related Webpages by Near-Duplicate Image
Retrieval
• C. Annotating Images by Mining Surrounding Text of
Related Webpages
IMAGE ANNOTATION SYSTEM
A. Constructing a Large-Scale Celebrity Name Vocabulary
1)Wikipedia
首段
信息框
标签
2)Entitycube
IMAGE ANNOTATION SYSTEM
• B. Discover Related Webpages by NearDuplicate Image Retrieval
– divide and conquer strategy
• 图片分成n×n
• 降维
• 阈值化
IMAGE ANNOTATION SYSTEM
• C. Annotating Images by Mining Surrounding
Text of Related Webpages
• 1) Type of names；
• 2) Type of surrounding text；
• 3) Frequency versus ratio；
MULTIMODAL NAME ASSIGNMENT
•
•
•
•
A. Notation
B. Overview of the Assignment Model
C. Label Propagation from SFSN Images p(Y|F)
D. Constrain the Propagation by a Context
Likelihood p(Y|T; λ)
• E. Normalization by Name Prior p(Y)
• F. Implementation Detail: Face Representation
• A. Notation
– faces in image In
– denote the face labels as
• B. Overview of the Assignment Model
– the confidence for label
• C. Label Propagation from SFSN Images p(Y|F)
– how to propagate labels from SFSN images to
SFMN and MF images
• D. Constrain the Propagation by a Context
Likelihood p(Y|T; λ)
• 1) For each image-level name vk, generate a
binary variable zk from p(vk |T) as defined in (3)
to indicate whether vk appears in image I.
• 2) If zk=1, generate mk faces of name vk in
image I from p(m|z; λ) as defined in (13).
• E. Normalization by Name Prior p(Y)
– p(Y) represents the prior of names.
• F. Implementation Detail: Face Representation
• the appearance of each face is described by local binary
pattern (LBP).
• the face image is divided into small regions from which
the LBP features are extracted and concatenated into a
single feature histogram.
• pply PCA to reduce the dimension of face descriptor
from over 3000 to 500 dimensions.
Evaluation