Parallelism for LDA
A uan, Changsi A
An Yang Ru
The probab
bility of the geenerated case can be calculaated by multiplyingg the joint prrobabilities off occurred evvents in the samplin
ng processes. (yang
gruan@indian
na.edu, anch@
@indiana.edu)
1.
Overview
w A
As parallelism is very importtant for large scale of data, we w
want to use different techn
nology to paraallelize the lattent d
dirichlet allocaation algorithm. Latent dirichlet allocattion (LLDA) is a generative m
model that allows sets of o
observations to
o be explained
d by unobservved groups wh
hich eexplain why so
ome parts of th
he data are sim
milar. TThere are two
o different im
mplementation
n of this model, o
one is Expectaation Maximum
m (EM metho
od) and the otther iss Gibbs Sampling. Both of them use iteraative algorithm
m to d
do the calculation, however, Gibbs Sampling is easier to im
mplement in ssequential cod
de but harderr to parallel while th
he EM method is easier to parallel as (Naallapati, Cohen
n et al.) has already use MapRed
duce technolo
ogy to paralleelize th
hat. And we are going to
o parallel thee Gibbs sampling im
mplementatio
on this time using iterative MapReduce and m
message passin
ng interface.
2.
Algorithm
m 2
2.1 Guideline o
of Sequential Algorithm with LDA LDA is a generative probab
bilistic model of a corpus. TThe b
basic idea is th
hat documentts are represeented as rand
dom m
mixtures over latent topics, where each topic is characterized by a distributtion over worrds(Blei, Ng ett al. 2
2003). TTo clarify thiss model, assume we havve the follow
wing convention: A
A corpus D : {d
d1, d2, .. dm}. EEach corpus di : {w1, w2, …, w
wni} iss a sequence of words. In
n contrast to words, we have vvocabulary V : {v1, v2, …, vl} which is aa non‐duplicaated p
projection of all words. Allso we assum
me there exisst K to
opics T = {t1, t2, …, tk}. A
Apart from thee above notation, we also h
have two variaable p
parameters: which is used as a param
meter in Dirich
hlet d
distribution to
o generate topics distribu
ution and is a conditional probability p
m
matrix, which records the p
probability of aa word given aa specific topicc. TThe algorithm with LDA mo
odel has two ggoals. One goaal is to
o generate a ttopics distribu
ution and asso
ociated wordss list fo
or each topiic, and give out the pro
obability of the ggenerated casee. Another goal is to updatee the parametters and in the generation prrocess. TThe algorithm can be desccribed in the following steeps: fiirst choose ∼
. Then for each word wi of d
document di , ssample a topicc Tn which follows multinom
mial d
distribution wiith parameterr , after thatt, sample a w
word wi’ according tto ( | Tn, )). After that, iff wi’ is not wi, we w
will update b
by augmentingg the probabillity of word wi’ to to
opic Tn and dim
minishing the one of word w
wi. 2.2 ACT mo
odel Author Co
onference Top
pic model (ACT) is an exttension model baseed on LDA(Tan
ng, Zhang et al. 2008). Because th
his model also
o adds conferrence and author as two new variables, so
o the size o
of the computation 2‐D matrixes. W
We will increased ffrom 2 2‐D matrixes to 3 2
have topiic‐author, wo
ord‐topic an
nd conferencce‐topic matrixes to do the Gibbs samplingg. And the saampled result (mattrix contains tthe frequencyy) can be used
d in the computatio
on to the final probability result:  is the probabilityy of a topic givven an authorr;  is the probability of a word
d given a to
opic;  is th
he probabilityy of a conferencee given a topicc. 3.
Techn
nology This time we are goingg to use an itterative MapReduce frameworkk (twister) and
d message passsing interfacee (MPI). The reason
n for choosing twister is thee algorithm is d
done in iterative p
pattern and it will be m
much faster to be parallelized
d under twisteer which is speecially designeed for it than normaal MapReducee framework like Hadoop. TThis has already been demonstraated by Jaliyaa et al. And M
MPI has parallelism method m
for yeears. By always be a standard p
ontrol group w
where there w
will be a using it we can have a co
n between thiis two technollogies.
comparison
Twister is an enhanceed MapReducce runtime w
with an extended programmingg model thatt supports itterative ons efficiently((J.Ekanayake, H.Li et MapReduce computatio
al. 2010). To use thaat feature, the iterative paart of the alggorithm needs to be done within the proggram control of the iteration. Here is a fiigure of the parallelism of the program. There are ttwo main partts of this proggram, the firstt part is the Gibbs sampling wh
hich is an iterative part of the nd the second
d none‐iterativve part is to u
use this program an
3 2‐D m
matrixes to calculate the final probability distribution
n. In the first part, for each mapper, theere will be 3 ssplit up 2‐D matrixxes in each mappers’ m
consstant memoryy which are generaated directly b
by each docum
ment chunk. In each itteration, the rreducer will broadcast the merged matrixes o
onto each map
pper. W
We call this method App
proximate Disstributed Autthor C
Conference To
opic model. (AD‐ACT) A
And Here is thee performance chart: Speeed up Test of 8
8k documents running on PolarGrid in Indiana University using 1 to 20 mappeers R
References: B
By using this p
pattern, we caan save the m
memory spacee on eeach mapper by exchange the time of sending the 1‐D m
matrix to each node. M
MPJ Express is a MPI extenssion of Java im
mplementation
n. It iss capable to rrun on distrib
buted machinees and multiccore p
processors. The usage of it is also daemo
on control sim
milar to
o Twister and with comman
nd line launcher. H
Here is a figuree and some AP
PI of it: P
P2P Communiccation C
Comm.Send() , Comm.Recv()) C
Comm.Isend(), Comm.Irecv()) C
Collective Communcation In
ntracomm.Bca
ast() In
ntracomm.Gather() P
Process Manag
gement In
ntra‐communiication(within
n same Group) : Intracomm In
nter‐communiication(server‐‐client, Carteesian topolog
gy): C
Cartcomm B
Blei, D. M., A. YY. Ng, et al. (2
2003). "Latent Dirichlet alloccation." Jo
ournal of Macchine Learning
g Research 3: 9
993‐1022. J..Ekanayake, H.Li, et al. (2010). Twister: A Runtim
me for itterative MapR
Reduce. Proceeedings of th
he First Intern
national W
Workshop on MapReduce a
and its Applica
ations of ACM
M HPDC 2
2010 conference June 20‐25,
5, 2010. Chicag
go, Illinois, AC
CM. N
Nallapati, R., W
W. Cohen, et al. Parallelized Variational EM for LLatent Dirichleet Allocation:: An Experim
mental Evalua
ation of SSpeed and Scalability. Pitttsburgh, USA
A, Carnegie Mellon U
University. TTang, J., J. Zhang, et al. (2
2008). ArnetM
Miner: Extractiion and M
Mining of Academic Social N
Networks. KD
DD’08, August 24–27, 2
2008, Las Vega
as, Nevada, USSA. 4.
on Conclusio
TThe performan
nce of Twister‐‐ACT is not as good as eexpected, sincee while we implement this parallel prograam, w
we found out tthat every tim
me we sample aa single elemeent in
nside the matrixes, there will be some inffluence to thee fo
ollowing samp
pling. So the reesults can nevver be as samee as th
he sequential one if we do parallel computing on the A
ACT ggibbs samplingg. Basically, to minimize thiss problem, we h
have to do iterrative mapreduce. And keep
p the each blo
ock o
of document o
on each mappeer’s memory ccache. Still wee h
have a large qu
uantity of messsage to pass ssince every m
mapper requires the full sam
mpling matrixees. Still, we can solve some of tthe memory p
problem by sp
plitting up the d
documents, so
o each mapperr only gets a partition of thee fu
ull document.