Attribute Learning for Understanding
Unstructured Social Activity
Yanwei Fu, Timothy M. Hospedales, Tao Xiang,
and Shaogang Gong
School of EECS, Queen Mary University of London, UK
Presented by Amr El-Labban
VGG Reading Group, Dec 5th 2012
Contributions
1.
Unstructured social activity attribute (USAA) dataset
2.
Semi-latent attribute space
3.
Topic model based attribute learning
Objective

Automatic classification of unstructured group social activity

Use an attribute based approach

Start with sparse, user defined attributes

Add latent ones

Learn jointly
Dataset

1500 videos, 8 classes

69 visual/audio attributes
manually labelled

Weak labelling

SIFT, STIP and MFCC
features used

Data available (features,
attributes, YouTube IDs)
Semi-Latent Attribute Space

Space consisting of:

User defined attributes

Discriminative latent attributes

Non-discriminative (background) latent
attributes
Topic modelling
𝑃 𝑥𝑖 𝑑𝑗 =
𝑘𝑃
𝑥𝑖 |𝑦𝑘 𝑃(𝑦𝑘 |𝑑𝑗 )
d
y
x
d
x
y
P(x|d)
=
x – low level features (‘words’)
y – attributes (‘topics’)
d – ‘documents’
P(x|y)
P(y|d)
Latent Dirichlet Allocation
y
x – low level features
y – attributes (user defined and latent)
θ – attribute distribution
φ – word distribution
α, β – Dirichlet parameters
x
Aside: Dirichlet disribution
 Distribution over multinomial distributions
 Parameterised by α
α = (6,2,2)
α = (2,3,4)
α = (3,7,5)
α = (6,2,6)
Aside: Dirichlet disribution

Important things to know:

α0 =
α𝑖
α𝑖
α0

𝐸 𝑋𝑖 =
- peak is closer to larger α values

𝑉𝑎𝑟 𝑋𝑖 =

α<1 gives more sparse distributions
α𝑖 (α0 −α𝑖 )
α0 2 (α0 +1)
- large α gives small variance
Latent Dirichlet Allocation
y
x – low level features
y – attributes (user defined and latent)
θ – attribute distribution
φ – word distribution
α, β – Dirichlet parameters
x
Latent Dirichlet Allocation
y
Generative model
for each document:
Choose θ ~ Dir(α)
Choose φ ~ Dir(β)
for each word:
Choose y ~ Multinomial(θ)
Choose x ~ Multinomial(φ y)
x
Latent Dirichlet Allocation
y
𝐾
𝑃 𝐷 𝛼, 𝛽 =
𝑀
𝑃(𝜑𝑘 |𝛽)
𝑘=1
x
𝑁
𝑃(𝜃𝑚 |𝛼)
𝑚=1
𝑃 𝑦𝑚,𝑛 𝜃𝑚 𝑃(𝑥𝑚,𝑛 |𝜑𝑦𝑚,𝑛 )
𝑛=1
Latent Dirichlet Allocation
y
x
 EM to learn Dirichlet parameters: α, β
 Approximate inference for posterior: 𝑃(θ, 𝑦 |𝑥, α, β)
SLAS


User defined part

Per instance prior on α.

Set to zero when attribute isn’t present in ground truth
Latent part


First half “class conditional”

One α per class.

All but one constrained to zero.
Second half “background”

Unconstrained
Classification

Use SLAS posterior to map from raw data to attributes

Use standard classifier (logistic regression) from attributes
to classes
N-shot transfer learning

Split data into two partitions – source and target

Learn attribute models on source data

Use N examples from target to learn attribute-class
mapping
Zero-shot learning



Detect novel class
Manually defined attribute-class “prototype”
Improve with self-training algorithm:
1.
Infer attributes for novel data
2.
NN matching in user defined space against protoype
3.
For each novel class:
4.
a)
Find top K matches
b)
Train new prototype in full attribute space (mean of top K)
NN matching in the full space
Experiments

Compare three models:

Direct: KNN or SVM on raw data

SVM-UD+LR: SVM to map raw data to attributes, LR maps attributes to
classes

SLAS+LR: SLAS to map raw data to attributes, LR learns classes based
on user-defined and class conditional attributes.
MASSIVE HACK

“The UD part of the SLAS topic profile is estimating the
same thing as the SVM attribute classifiers, however the
latter are slightly more reliable due to being discriminatively
optimised. As input to LR, we therefore actually use the
SVM attribute classier outputs in conjunction with the latent
part of our topic profile.”
Results - classification

SLAS+LR better as number if training data and user defined
attributes decreases

Copes with 25% wrong attribute bits
Results - classification

KNN and SVM have vertical bands – consistent
misclassification
Results – N-shot transfer learning

Vary number of user defined attributes

SVM+LR cannot cope with zero attributes
Results – Zero-shot transfer learning


Two cases:

Continuous prototype – mean attribute profile

Binary prototype – thresholded mean
Tested without background latent attributes (SLAS(NF))
Conclusion

Augmenting SVM and user defined attributes with latent
ones definitely helps.

Experimental hacks make it hard to say how good the model
really is…