Outline
A Matrix Factorization Framework for
JJointlyy Analyzing
y g Multiple
p Nonnegative
g
Data Sources
Sunil Kumar Gupta, Dinh Phung*,
Phung*, Brett Adams, Svetha Venkatesh





Motivation
Shared Subspace Learning
Applications
Experimental Results
Conclusion
Institute for MultiMulti-sensor Processing & Content Analysis (IMPCA)
Curtin University of Technology, Perth, Australia
Text Mining Workshop 2011, Arizona, USA
30th April, 2011
* Presenting author
Problem
Aim
Joint modelling of multiple data sources to exploit their collective strength while
retaining their variability or differences.
M ti ti
Motivation




Research community has mainly focused their efforts on analyzing single
data source.
Subspace learning across multiple data sources can capture certain
information otherwise not possible if analyzing them independently.
CCA based methods can’t be applied in general scenarios due to the need
for correspondences across each data source
source.
Can we develop a model which can systematically capture the collective
strengths of the data from multiple sources which share some underlying
structures?
(*Ack: some graphics are from images.google.com)
Can we tell what is going on: jointly and individually?
Nonnegative Data Sources
Multiple Nonnegative Shared
S b
Subspace
Learning
i ((MS(MS
S-NMF))
In this work, we confine ourselves to modelling only nonnegative data sources.
g
data sources are important
p
and widelyy encountered in real world such as
Nonnegative




Text
Image
Video
Counting data
NMF
Multiple Nonnegative Shared
S b
Subspace
Learning
i ((MS(MS
S-NMF))
Multiple Nonnegative Shared
S b
Subspace
Learning
i ((MS(MS
S-NMF))
Let us denote the data matrix for i-th source by Xi and its dimension be MxNi and
write
it the
th decomposition
d
iti as :
where
S n, i   v  S n  | i  v
and
S n   PowerSet1,..., n
For n = 3 data sources :
X 1  [W1 | W12 | W13 | W123 ][ H1T,1 | H1T,12 | H1T,13 | H1T,123 ]T
(a) Chain sharing
(b) Pair-wise
Pair wise sharing
(c) Full sharing
X 2  [W2 | W12 | W23 | W123 ][ H 2T, 2 | H 2T,12 | H 2T, 23 | H 2T,123 ]T
X 3  [W3 | W13 | W23 | W123 ][ H 3T,3 | H 3T,13 | H 3T, 23 | H 3T,123 ]T
 We have freedom to specify what sharing configuration to be used.
Multiplicative updates
…(MS--NMF continued)
…(MS
We minimize the following joint decomposition error computed across all data matrices
where ||.||F is the Frobenius norm and λi is defined as the following
We propose iterative solution for the above problem and details can be found in the paper.
Social Media Applications
Social Media Retrieval
MS-NMF based retrieval algorithm


Improving social media retrieval in target medium with
the
h hhelp
l off other
h auxiliary
ili
social
i l media
di sources.
Cross-media retrieval or retrieval across multiple social
Crossmedia sources.
W , H 
v
Query set
(Q)
Vocabulary
(V)
{Retrieved
items}
form query vector qx
using vocabulary V and Q
rank the similarities in
decreasing order
No. of items to be
retrieved (N)
i ,v
project qx onto the subspace (to get qh)
q x  Wi qh
compute cosine similarity between
query
q
y vector and the items in the
subspace Wi
Cross--Social Media Retrieval
Cross
MS-NMF based cross-media retrieval algorithm
Vocabulary
(V)
{Retrieved
items}
form query vector qx
using vocabulary V and Q
Data collection
W , H 
Use subspace Wv for cross media configuration v,
v e.g.
e g W12 for retrieval across
medium 1 and 2, similarly, use W123 for retrieval across medium 1, 2 and 3.
Query set
(Q)
Experiments
v
We created a cross social media data by crawling the textual tags of three disparate
social media genres :
 Text (from BlogSpot website)
 Image (from Flickr website)
 Video (from YouTube website)
i ,v
project qx onto the subspace (to get qh)
q x  Wv qh
Dataset
size
compute cosine similarity b/w
qh and the items of involved media
(e g Hi,v and Hj,v) in the
(e.g.
subspace Wv
rank the similarities in
decreasing order
Data Set : Concept Distribution
Christmas
Holi
Academy Awards
Australian Open
Olympic Games
US Election
El i
Earthquake
Terror Attacks
Global Warming
Concepts
Avg. Tags
Per Item (rounded
rounded))
BlogSpot
10000
‘Academy Awards’, ‘Australian Open’, ‘Olympic Games’, ‘US
Election’, ‘Christmas’, ‘Earthquake’, ‘Cricket World Cup’
6
Flickr
20000
‘Academy Awards’, ‘Australian Open’, ‘Olympic Games’, ‘US
Election’, ‘Christmas’, ‘Terror Attacks’ , ‘Holi’
8
YouTube
7000
‘Academy Awards’, ‘Australian Open’, ‘Olympic Games’, ‘US
Election’, ‘Terror
Election
Terror Attacks
Attacks’, ‘Earthquake’
Earthquake , ‘Global
Global Warming
Warming’
7
Choice Subspace Dimensions (Kv)

Find the number of the common features (tags in our case) between the two datasets, say Mv.

Use “the
the rule of thumb
thumb” suggested by [K.V.
[K V Mardia et al 1979,
1979 Multivariate Analysis]
Analysis] as
Kv  M v / 2

Initialize using above heuristic and then perform cross-validation based on retrieval precision
performance.
Experiment--I
Experiment
Experiment--II
Experiment
(Improving Social Media Retrieval in Transfer Learning Setting)
(Retrieving Items across Multiple Social Media Sources)
BASELINES NMF (no sharing), JSNMF [7] with BlogSpot as auxiliary,
JSNMF[7] with Flickr,
Flickr and tag-based
Precision and Recall measures for cross-media scenario are defined as the following:
where n is the number of media involved for retrieval ; for a particular query, Ai and
Gi are the answer set and ground-truth set from the i-th medium.
Baseline-I : Tag based matching
BASELINES : Baseline-II : Lin et al. [12]
Baseline-III : JSNMF [7]
Precision-Scope and MAP plots
11-point Precision-Recall plots
Cross-media retrieval results across
BlogSpot/Flickr
/
11-point Precision-Recall (BlogSpot/Flickr)
Precision-Scope and MAP (BlogSpot/Flickr)
Cross-media retrieval results across
BlogSpot/YouTube
/
b
11-point Precision-Recall (BlogSpot/YouTube)
Precision-Scope and MAP (BlogSpot/YouTube)
Cross-media retrieval results across
Flickr/YouTube
/
b
Cross-media retrieval results across
BlogSpot/Flickr/YouTube
/
/
b
11-point Precision-Recall
Precision-Scope and MAP
JSNMF[7] can not be applied in this case as it is limited to two data source cases only !
11-point Precision-Recall (Flickr/YouTube)
Precision-Scope and MAP (Flickr/YouTube)
Topical
p
Analysis
y
Conclusion
Definition of Entropy is usual whereas Impurity of a topic is defined as the following
where NGD(tx, ty) is normalized Google Distance [4] between two terms tx and ty;
g
words in a topic.
p
and L is the number of “significant”
Distribution of Entropy and Impurity values computed across various topics
(a) Entropy Distribution
(b) Impurity Distribution

We presented a novel framework for jointly modelling data from multiple
nonnegative sources with arbitrary sharing topologies.
topologies

We demonstrated its application on two social media problems (1) improved tagtagbased social media retrieval within one domain (2) CrossCross-social media retrieval

We empirically demonstrated that controlled sharing is crucial to avoid any
negative knowledgeknowledge-transfer from auxiliary data sources.

Our MSMS-NMF framework is generic and can be applied to used to exploit sharing
strengths of multiple data sources.
References
1.
2
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
R.K. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. The Journal of Machine Learning Research,
6:1817–1853, 2005.
R B
R.
Baeza-Yates,
z Y t B.
B Ribeiro-Neto,
Rib ir N t ett al.l Modern
M d iinformation
f
ti retrieval.
t i l Addison-Wesley
Addi
W l R
Reading,
di MA
MA, 1999
1999.
M.W. Berry and M. Browne. Email surveillance using nonnegative matrix factorization. Computational & Mathematical Organization Theory, 11(3):249–264,
2005.
R.L. Cilibrasi, P.M.B. Vitanyi, and A. CWI. The google similarity distance. IEEE Transactions on Knowledge and Data Engineering, 19(3):370–383, 2007.
S.A. Golder and B.A. Huberman. Usage patterns of collaborative tagging systems. Journal of Information Science, 32(2):198, 2006.
Q. Gu and J. Zhou. Learning the shared subspace for multi-task clustering and transductive transfer classification. ICDM, pages 159–168, 2009.
S.K. Gupta, D. Phung, B. Adams, T. Tran, and S. Venkatesh. Nonnegative shared subspace learning and its application to social media retrieval. SIGKDD,
pages 1169–1178,
1169 11 8 2010.
2010
S. Ji, L. Tang, S. Yu, and J. Ye. A shared-subspace learning framework for multi-label classification. ACM Transactions on Knowledge Discovery from Data, 4(2):1–
29, 2010.
M.S. Kankanhalli and Y. Rui. Application potential of multimedia information retrieval. Proceedings of the IEEE, 96(4):712–720, 2008.
D.D. Lee and H.S. Seung. Algorithms for non-negative matrix factorization. Advances in Neural Information Processing Systems, 13, 2001.
C.J. Lin. Projected gradient methods for nonnegative matrix factorization. Neural Computation, 19(10):2756–2779, 2007.
Y.R. Lin,, H. Sundaram,, M. De Choudhury,
y, and A. Kelliher. Temporal
p
patterns
p
in social media streams: Theme discoveryy and evolution usingg joint
j
analysis
y
of content and context. In ICME, pages 1456–1459, 2009.
K. V. Mardia, J. M. Bibby, and J. T. Kent. Multivariate analysis. Academic Press, New York, 1979.
C. Marlow, M. Naaman, D. Boyd, and M. Davis. Ht06, tagging paper, taxonomy, flickr, academic article, toread. Proceedings Hypertext, pages 31–40, 2006.
F. Shahnaz, M.W. Berry, V.P. Pauca, and R.J. Plemmons. Document clustering using nonnegative matrix factorization. Information Processing and Management,
42(2):373–386, 2006.
S. Si, D. Tao, and B. Geng. Bregman divergence based regularization for transfer subspace learning. IEEE Transactions on Knowledge and Data Engineering,
( )
, 2009.
22(7):929–942,
B. Sigurbj¨ornsson and R. Van Zwol. Flickr tag recommendation based on collective knowledge. WWW, pages 327–336, 2008.
Wei Xu, Xin Liu, and Yihong Gong. Document clustering based on non-negative matrix factorization. SIGIR, pages 267–273, 2003.
R. Yan, J. Tesic, and J.R. Smith. Model-shared subspace boosting for multi-label classification. SIGKDD, pages 834–843, 2007.
Y. Yang, D. Xu, F. Nie, J. Luo, and Y. Zhuang. Ranking with local regression and global alignment for cross media retrieval. MM, pages 175–184, 2009.
Y. Yi, Y.T. Zhuang, F. Wu, and Y.H. Pan. Harmonizing hierarchical manifolds for multimedia document semantics understanding and cross-media retrieval.
Language, 1520:9210, 2008.