Informatics and Mathematical Modelling / Intelligent Signal Processing
Extensions of Non-negative
Matrix Factorization to Higher
Order data
Morten Mørup
Informatics and Mathematical Modeling
Intelligent Signal Processing
Technical University of Denmark
Morten Mørup
1
Informatics and Mathematical Modelling / Intelligent Signal Processing
Sæby, May 22-2006
Parts of the work done in
collaboration with
Lars Kai Hansen, Professor
Sidse M. Arnfred, Dr. Med. PhD
Mikkel N. Schmidt, Stud. PhD
Department of Signal Processing
Informatics and Mathematical Modeling,
Technical University of Denmark
Cognitive Research Unit
Hvidovre Hospital
University Hospital of Copenhagen
Department of Signal Processing
Informatics and Mathematical Modeling,
Technical University of Denmark
Morten Mørup
2
Informatics and Mathematical Modelling / Intelligent Signal Processing
Outline
 Non-negativity Matrix Factorization
(NMF)
 Sparse coding
(SNMF)
 Convolutive PARAFAC models (cPARAFAC)
 Higher Order Non-negative Matrix Factorization
(an extension of NMF to the Tucker model)
Morten Mørup
3
Informatics and Mathematical Modelling / Intelligent Signal Processing
NMF is based on Gradient Descent
NMF:
VWH s.t. Wi,d,Hd,j0
Let C be a given cost function, then update the
parameters according to:
Morten Mørup
4
Informatics and Mathematical Modelling / Intelligent Signal Processing
The idea behind multiplicative updates
Positive term
Morten Mørup
5
Negative term
Informatics and Mathematical Modelling / Intelligent Signal Processing
Non-negative matrix factorization (NMF)
(Lee & Seung - 2001)
NMF gives Part based representation
(Lee & Seung – Nature 1999)
Morten Mørup
6
Informatics and Mathematical Modelling / Intelligent Signal Processing
The NMF decomposition is not unique
Simplical Cone
~~
V  WH  (WP)(P -1 H)  WH
NMF only unique when data adequately spans the positive orthant
(Donoho & Stodden - 2004)
Morten Mørup
7
Informatics and Mathematical Modelling / Intelligent Signal Processing
Sparse Coding NMF (SNMF)
(Eggert & Körner, 2004)
Morten Mørup
8
Informatics and Mathematical Modelling / Intelligent Signal Processing
Swimmer Articulations
Illustration (the swimmer problem)
V ( Articulation pixel )  W ( ArticulationExpression) H ( Expression pixel )
True Expressions
Morten Mørup
NMF Expressions
9
SNMF Expressions
Informatics and Mathematical Modelling / Intelligent Signal Processing
Why sparseness?
 Ensures uniqueness
 Eases interpretability
(sparse representation  factor effects pertain to fewer dimensions)
 Can work as model selection
(Sparseness can turn off excess factors by letting them become zero)
 Resolves over complete representations
(when model has many more free variables than data points)
Morten Mørup
10
Informatics and Mathematical Modelling / Intelligent Signal Processing
PART I: Convolutive PARAFAC (cPARAFAC)
Morten Mørup
11
Informatics and Mathematical Modelling / Intelligent Signal Processing
By cPARAFAC means PARAFAC convolutive in at
least one modality
Vi , j ,k   Di , W j , H k ,



Regular PARAFAC
Vi , j ,k   Di , W j  , Hk ,



cPARAFAC
Vi , j ,k   Di , Wj  , Hk  ,



X i , j   Ai , S j  ,

Convolution: The process of generating X
by convolving (sending) the sources S
through the filter A
Deconvolution: The process of estimating
the filter A from X and S
c2PARAFAC
Convolution can be in any combination of modalities
-Single convolutive, double convolutive etc.
Morten Mørup
12
Informatics and Mathematical Modelling / Intelligent Signal Processing
Relation to other models
 PARAFAC2 (Harshman, Kiers, Bro)
 Shifted PARAFAC (Hong and Harshman, 2003)
Vi , j ,k   Di , Wj  , Hk ,

3
3
cPARAFAC can account for echo effects
Morten Mørup
13
cPARAFAC becomes shifted PARAFAC
when convolutive filter is sparse
Informatics and Mathematical Modelling / Intelligent Signal Processing
Application example of cPARAFAC
Transcription and separation of music
Morten Mørup
14
Informatics and Mathematical Modelling / Intelligent Signal Processing
The ‘ideal’ Log-frequency Magnitude Spectrogram
of an instrument
 Different notes played by an
instrument corresponds on a
logarithmic frequency scale to a
translation of the same harmonic
structure of a fixed temporal pattern
1600
800
Mozart Sonate no,. 16 in C Major
Frequency [Hz]
3200
400
200
0
0.5
Morten Mørup
1
1.5
2
Time [s]
2.5
3
Tchaikovsky: Violin Concert in D Major
3.5
15
Informatics and Mathematical Modelling / Intelligent Signal Processing
NMF 2D deconvolution (NMF2D1): The Basic Idea
 Model a log-spectrogram of polyphonic music by an
extended type of non-negative matrix factorization:
– The frequency signature of a specific note played by an
instrument has a fixed temporal pattern (echo)
 model convolutive in time
– Different notes of same instrument has same time-logfrequency signature but varying in fundamental frequency
(shift)
 model convolutive in the log-frequency axis.
(1Mørup & Scmidt, 2006)
Morten Mørup
16
Informatics and Mathematical Modelling / Intelligent Signal Processing
NMF2D Model
 NMF2D Model – extension of NMFD1:
(1Smaragdis, 2004, Eggert et al. 2004, Fitzgerald et al. 2005)
Morten Mørup
17
Informatics and Mathematical Modelling / Intelligent Signal Processing
8
4
0

Understanding the NMF2D Model
1600
800
400
200
0246

Morten Mørup
0
0.2
0.4
0.6
Time [s]
18
0.8
Frequency [Hz]
3200
Informatics and Mathematical Modelling / Intelligent Signal Processing
The NMF2D has inherent ambiguity between the
structure in W and H
To resolve this ambiguity sparsity is imposed
on H to force ambiguous structure onto W
Morten Mørup
19
Informatics and Mathematical Modelling / Intelligent Signal Processing
Real music example of how imposing sparseness
resolves the ambiguity between W and H
NMF2D
Morten Mørup
20
SNMF2D
Informatics and Mathematical Modelling / Intelligent Signal Processing
Extension to multi channel analysis by the
PARAFAC model
bλ
Factor analysis
(Charles Spearman ~1900)
Morten Mørup
PARAFAC
(Harshman & Carrol and Chang 1970)
21
Informatics and Mathematical Modelling / Intelligent Signal Processing
cPARAFAC: Sparse Non-negative Tensor Factor 2D
deconvolution (SNTF2D)
(Extension of Fitzgerald et al. 2005, 2006 to form a sparse double deconvolution)
Morten Mørup
22
Informatics and Mathematical Modelling / Intelligent Signal Processing
SNTF2D algorithms
Morten Mørup
23
Informatics and Mathematical Modelling / Intelligent Signal Processing
Mozart Sonate no. 16 in C Major


Tchaikovsky: Violin Concert in D Major

Morten Mørup

24
Informatics and Mathematical Modelling / Intelligent Signal Processing
Stereo recording of ”Fog is Lifting” by Carl Nielsen
Stereo Channel 2
Stereo Channel 1
Log-Spectrogram Channel 1
Log-Spectrogram Channel 2
22 kHz
50 Hz
50 Hz
0.9071
25.9 ms
0.420
6850
Estimated Harp
22 kHz
22 kHz
50 Hz
50 Hz
25.9 ms
Morten Mørup
9
25.9 ms
0.7286
22 kHz
Estimated Flute
25.9 ms
25
Informatics and Mathematical Modelling / Intelligent Signal Processing
Applications
 Applications
–
–
–
–
Source separation.
Music information retrieval.
Automatic music transcription (MIDI compression).
Source localization (beam forming)
Morten Mørup
26
Informatics and Mathematical Modelling / Intelligent Signal Processing
PART II: Higher Order NMF (HONMF)
Morten Mørup
27
Informatics and Mathematical Modelling / Intelligent Signal Processing
Higher Order Non-negative Matrix Factorization
(HONMF)
Motivation:
Many of the data sets previously explored by the Tucker model are
non-negative and could with good reason be decomposed under
constraints of non-negativity on all modalities including the core.
BatchSpectreTime
X Strength
(Smilde et al. 1999,2004, Andersson & Bro 1998, Nørgard & Ridder 1994)
 Spectroscopy data
 Web mining
UsersQueriesWeb pages
X Click
counts
 Image Analysis
PeopleViewsIlluminationsExpressionsPixels
X Image
Intensity
 Semantic Differential Data
Judges Music PiecesScales
X Grade
(Sun et al., 2004)
(Vasilescu and Terzopoulos, 2002, Wang and Ahuja, 2003, Jian and Gong, 2005)
(Murakami and Kroonenberg, 2003)
 And many more……
Morten Mørup
28
Informatics and Mathematical Modelling / Intelligent Signal Processing
However, non-negative Tucker decompositions are not
in general unique!
But - Imposing sparseness overcomes this problem!
Morten Mørup
29
Informatics and Mathematical Modelling / Intelligent Signal Processing
The Tucker Model
Morten Mørup
30
Informatics and Mathematical Modelling / Intelligent Signal Processing
Algorithms for HONMF
Morten Mørup
31
Informatics and Mathematical Modelling / Intelligent Signal Processing
Results
HONMF with sparseness, above imposed on the core can
be used for model selection -here indicating the PARAFAC
model is the appropriate model to the data.
Furthermore, the HONMF gives a more part based hence easy
interpretable solution than the HOSVD.
Morten Mørup
32
Informatics and Mathematical Modelling / Intelligent Signal Processing
Evaluation of uniqueness
Morten Mørup
33
Informatics and Mathematical Modelling / Intelligent Signal Processing
Data of a Flow Injection Analysis (Nørrgaard, 1994)
HONMF with sparse core and mixing captures unsupervised
the true mixing and model order!
Morten Mørup
34
Informatics and Mathematical Modelling / Intelligent Signal Processing
Conclusion
 HONMF not in general unique, however when
imposing sparseness uniqueness can be achieved.
 Algorithms devised for LS and KL able to impose
sparseness on any combination of modalities
 The HONMF decompositions more part based hence
easier to interpret than other Tucker decompositions
such as the HOSVD.
 Imposing sparseness can work as model selection
turning of excess components
Morten Mørup
35
Informatics and Mathematical Modelling / Intelligent Signal Processing
Coming soon in a MATLAB implementation near You
Morten Mørup
36
Informatics and Mathematical Modelling / Intelligent Signal Processing
References
Carroll, J. D. and Chang, J. J. Analysis of individual differences in multidimensional scaling via an N-way generalization of "Eckart-Young" decomposition, Psychometrika 35 1970 283--319
Eggert, J. and Korner, E. Sparse coding and NMF. In Neural Networks volume 4, pages 2529-2533, 2004
Eggert, J et al Transformation-invariant representation and nmf. In Neural Networks, volume 4 , pages 535-2539, 2004
Fiitzgerald, D. et al. Non-negative tensor factorization for sound source separation. In proceedings of Irish Signals and Systems Conference, 2005
FitzGerald, D. and Coyle, E. C Sound source separation using shifted non.-negative tensor factorization. In ICASSP2006, 2006
Fitzgerald, D et al. Shifted non-negative matrix factorization for sound source separation. In Proceedings of the IEEE conference on Statistics in Signal Processing. 2005
Harshman, R. A. Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-modal factor analysis},UCLA Working Papers in Phonetics 16 1970 1—84
Harshman, Richard A.Harshman and Hong, Sungjin Lundy, Margaret E. Shifted factor analysis—Part I: Models and properties J. Chemometrics (17) pages 379–388, 2003
Kiers, Henk A. L. and Berge, Jos M. F. ten and Bro, Rasmus PARAFAC2 - Part I. A direct fitting algorithm for the PARAFAC2 model, Journal of Chemometrics (13) nr.3-4 pages 275-294, 1999
Lathauwer, Lieven De and Moor, Bart De and Vandewalle, Joos MULTILINEAR SINGULAR VALUE DECOMPOSITION.SIAM J. MATRIX ANAL. APPL.2000 (21)1253–1278
Lee, D.D. and Seung, H.S. Algorithms for non-negative matrix factorization. In NIPS, pages 556-462, 2000
Lee, D.D and Seung, H.S. Learning the parts of objects by non-negative matrix factorization, NATURE 1999
Murakami, Takashi and Kroonenberg, Pieter M. Three-Mode Models and Individual Differences in Semantic Differential Data, Multivariate Behavioral Research(38) no. 2 pages 247-283, 2003
Mørup, M. and Hansen, L.K.and Arnfred, S.M.Decomposing the time-frequency representation of EEG using nonnegative matrix and multi-way factorization Technical report, Institute for Mathematical
Modeling, Technical University of Denmark, 2006a
Mørup, M. and Schmidt, M.N. Sparse non-negative matrix factor 2-D deconvolution. Technical report, Institute for Mathematical Modeling, Tehcnical University of Denmark, 2006b
Mørup, M and Schmidt, M.N. Non-negative Tensor Factor 2D Deconvolution for multi-channel time-frequency analysis. Technical report, Institute for Mathematical Modeling, Technical University of
Denmark, 2006c
Schmidt, M.N. and Mørup, M. Non-negative matrix factor 2D deconvolution for blind single channel source separation. In ICA2006, pages 700-707, 2006d
Mørup, M. and Hansen, L.K.and Arnfred, S.M. Algorithms for Sparse Higher Order Non-negative Matrix Factorization (HONMF), Technical report, Institute for Mathematical Modeling, Technical
University of Denmark, 2006e
Nørgaard, L and Ridder, C.Rank annihilation factor analysis applied to flow injection analysis with photodiode-array detection Chemometrics and Intelligent Laboratory Systems 1994 (23) 107-114
Schmidt, M.N. and Mørup, M. Sparse Non-negative Matrix Factor 2-D Deconvolution for Automatic Transcription of Polyphonic Music, Technical report, Institute for Mathematical Modelling, Tehcnical
University of Denmark, 2005
Smaragdis, P. Non-negative Matrix Factor deconvolution; Extraction of multiple sound sources from monophonic inputs. International Symposium on independent Component Analysis and Blind Source
Separation (ICA)W
Smilde, Age K. Smilde and Tauller, Roma and Saurina, Javier and Bro, Rasmus, Calibration methods for complex second-order data Analytica Chimica Acta 1999 237-251
Sun, Jian-Tao and Zeng, Hua-Jun and Liu, Huanand Lu Yuchang and Chen Zheng CubeSVD: a novel approach to personalized Web search WWW '05: Proceedings of the 14th international conference on
World Wide Web pages 382—390, 2005
Tamara G. Kolda Multilinear operators for higher-order decompositions technical report Sandia national laboratory 2006 SAND2006-2081.
Tucker, L. R. Some mathematical notes on three-mode factor analysis Psychometrika 31 1966 279—311
Welling, M. and Weber, M. Positive tensor factorization. Pattern Recogn. Lett. 2001
Vasilescu , M. A. O. and Terzopoulos , Demetri Multilinear Analysis of Image Ensembles: TensorFaces, ECCV '02: Proceedings of the 7th European Conference on Computer Vision-Part I, 2002
Morten Mørup
37