LATENT DIRICHLET
ALLOCATION
Outline
• Introduction
• Model Description
• Inference and Parameter Estimation
• Example
• Reference
Introduction
As more information becomes available, it becomes more difficult
to access what we are looking for.
We need new tools to help us organize, search, and understand
these vast amounts of information.
Introduction
Topic modeling provides methods for automatically organizing,
understanding, searching, and summarizing large electronic archives.
• Uncover the hidden topical patterns that pervade the collection.
• Annotate the documents according to those topics.
• Use the annotations to organize, summarize, and search the texts.
Intuition behind LDA
P(b ,q , z,w)
Notation and Assumption
• We have a set of documents D , D
corpus.
1
2
,..., DM, constituting a
• Each document is a collection of words or a “bag of
words”. (Exchangeability)
• After elimination of some stopping words, a corpus
contains V words: w1,w2 ,..., wV , involve K topic with
distributions: b1,..., b k
• Each document is composed of N “important” or
“Effective” words: w1,w2 ,..., wN and with topic
proportions q d .
1….. topic …..K
1...nth word..Nd
1…word idx…V
Model Definition
(bd | h ) ~ Dir(h ) (q d | a ) ~ Dir(a ) Zd,n ~ Multi(q d ) Wd,n ~ Multi(bz ,n )
d
Dirichlet and Multinomial Distribution
• It’s more like such a distribution that is used to describe
another distribution. E.g. Multinomial
• Multinomial:
n!
P(X1 = x1,..., XK = xK ) =
p1x1 ...pKxK
x1 !...xK !
K
where X Î{0,..., n} and
å Xi = n
i
i=1
K
• Dirichlet
G(å a i )
P(q | a ) = K i=1
q1a1 -1 ...q Ka K -1
Õ G(a i )
qi > 0
åq
i
=1
i=1
Where variable \theta can take values in the (k-1) simplex.
Dirichlet and Multinomial Distribution
Properties
LSA & LDA
Reference
• Latent Dirichlet Allocation, DM Blei, AY Ng, MI jordan –
the journal of machine learning research, 2003
• Topic Models Vs. Unstructured Data, G Anthes –
Communications of the ACM, 2010
• Probabilistic Topic Models, M Steyvers, T Griffiths –
Handbook of latent sematic analysis, 2007
• GibbsSampling for the Uninitiated, P Resnik, E
Hardisty - 2010