Bayesian Nonparametric Matrix Factorization for
Recorded Music
Authors: Matthew D. Hoffman, David M. Blei, Perry R. cook
Princeton University, Department of Computer Science,
35 olden St., Princeton, NJ, 08540 USA
Reading Group Presenter:
Shujie Hou
Cognitive Radio Institute
Friday, October 15, 2010
Outline
■ Introduction
■ Terminology
■ Problem statement and contribution of this paper
■ Gap-NMF Model(Gamma Process Nonnegative Matrix Factorization )
■ Variational Inference
■ Definition
■ Variational Objective Function
■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
Terminology(1)
■ Nonparametric Statistics:
□ The term non-parametric is not meant to imply that such models
completely lack parameters but that the number and nature of
the parameters are flexible and not fixed in advance.
■ Nonnegative Matrix Factorization:
□ Non-negative matrix factorization (NMF) is a group of algorithms
in multivariate analysis and linear algebra where a matrix, is
factorized into (usually) two matrices with all elements are
greater than or equal to 0
X  WH
The above two definitions are cited from Wikipedia
Terminology(2)
■ Variational Inference:
□ Variational inference approximates the posterior distribution with
a simpler distribution, whose parameters are optimized to be
close to the true posterior.
■ Mean-field Variational Inference:
□ In mean-field variational inference, each variable is given an
independent distribution, usually of the same family as its prior.
Outline
■ Introduction
■ Terminology
■ Problem statement and Contribution of this Paper
■ Gap-NMF Model
■ Variational Inference
■ Definition
■ Variational Objective Function
■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
Problem Statement and Contribution
■ Research Topic:
□ Breaking audio spectrograms into separate sources of sound
using latent variable decompositions. E.g., matrix factorization.
■ A potential problem :
□ The number of latent variables must be specified in advance
which is not always possible.
■ Contribution of this paper
□ The paper develops Gamma Process Nonnegative Matrix
Factorization (GaP-NMF), a Bayesian nonparametric approach
to decompose spectrograms.
Outline
■ Introduction
■ Terminology
■ Problem statement and Contribution of this Paper
■ Gap-NMF Model
■ Variational Inference
■ Definition
■ Variational Objective Function
■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
Dataset on GaP-NMF Model
■ What are given is a M by N matrix X
in which Xmn is the power of audio signal at time window n and
frequency bin m.
If the number of latent variable is specified in advance:
■ Assuming the audio signal is composed of K static sound
sources. The problem is to decompose X  WH
, in which W
is M by K matrix, H is K by N matrix. In which cell Wmk is the
average amount of energy source k exhibits at frequency m.
cell H kn is the gain of source k at time n.
■ The problem is solved by
GaP-NMF Model
If the number of latent variable is not specified in advance:
■ GaP-NMF assumes that the data is drawn according to the
following generative process:
Based on the formula that
(Abdallah&Plumbley (2004))
GaP-NMF Model
If the number of latent variable is not specified in advance:
■ GaP-NMF assumes that the data is drawn according to the
following generative process:
The overall gain of the
corresponding source l
Used to control the
number of latent
variables
Based on the formula that
(Abdallah&Plumbley (2004))
GaP-NMF Model
Kingman ,1993
■ The number of nonzero
is the number of the latent
variables K.
■ If L increased towards infinity, the nonzero L which expressed
by K is finite and obeys:
Outline
■ Introduction
■ Terminology
■ Problem statement and Contribution of this Paper
■ Gap-NMF Model
■ Variational Inference
■ Definition
■ Variational Objective Function
■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
Definition of Variational Inference
■ Variational inference approximates the posterior distribution with a
simpler distribution, whose parameters are optimized to be close to
the true posterior.
■ Under this paper’s condition:
Posterior Distribution
What measured
Definition of Variational Inference
■ Variational inference approximates the posterior distribution with a
simpler distribution, whose parameters are optimized to be close to
the true posterior.
■ Under this paper’s condition:
Variational Distribution
Posterior Distribution
Approximates
Variational distribution
assumption with free
parameters
What measured
Definition of Variational Inference
■ Variational inference approximates the posterior distribution with a
simpler distribution, whose parameters are optimized to be close to
the true posterior.
■ Under this paper’s condition:
Variational Distribution
Adjust Parameters
Posterior Distribution
Approximates
Variational distribution
assumption with free
parameters
What measured
Outline
■ Introduction
■ Terminology
■ Problem statement and Contribution of this Paper
■ Gap-NMF Model
■ Variational Inference
■ Definition
■ Variational Objective Function
■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
Variational Objective Function
■ Assume each variable obeys the following Generalized
Inverse-Gaussian (GIG) family:
Variational Objective Function
■ Assume each variable obeys the following Generalized
Inverse-Gaussian (GIG) family:
It is Gamma
family
Variational Objective Function
■ Assume each variable obeys the following Generalized
Inverse-Gaussian (GIG) family:
It is Gamma
family
Denotes a modified
Bessel function of the
second kind
Deduction(1)
From Jordan
et al., 1999
■ The difference between the left and right sides is the
Kullback-Leibler divergence between the true posterior and
the variational distribution q.
■ Kullback-Leibler divergence : for probability distributions P
and Q of a discrete random variable their K–L divergence is
defined to be
Deduction(2)
Deduction(2)
Using Jensen’s inequality
Objective function
■ L=
Bounded by
■ The objective function becomes
+
■ Maximize the objective function defined above with the
corresponding parameters.
■ The distribution is obtained:
■ Because these three distributions are independent, we gain
approximates
Outline
■ Introduction
■ Terminology
■ Problem statement and Contribution of this Paper
■ Gap-NMF Model
■ Variational Inference
■ Definition
■ Variational Objective Function
■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
Coordinate Ascent Algorithm(1)
■ The derivative of the objective function with respect to
variational parameters equals to zero to obtain:
■ Similarly:
Coordinate Ascent Algorithm(2)
■ Using Lagrange multipliers, then the bound parameters
become
■ Then updating bound parameters and variational parameters
according to equations 14,15,16,17 and18 to ultimately
reaching a local minimum.
Outline
■ Introduction
■ Terminology
■ Problem statement and Contribution of this Paper
■ Gap-NMF Model
■ Variational Inference
■ Definition
■ Variational Objective Function
■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
Other Approaches
■
■
■
■
Finite Bayesian Model ( also called GIG-NMF).
Finite Non-Bayesian Model.
EU-Nonnegative Matrix Factorization.
KL-Nonnegative Matrix Factorization.
Outline
■ Introduction
■ Terminology
■ Problem statement and Contribution of this Paper
■ Gap-NMF Model
■ Variational Inference
■ Definition
■ Variational Objective Function
■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
Evaluation on Synthetic data(1)
■ The data is generated according to the following model:
Evaluation on Synthetic data(2)
Evaluation on Recorded Music
Conclusion
■ Gap-NMF model is capable of determining the number of
latent source automatically.
■ The key step of the paper is to use variational distribution to
approximate posterior distribution.
■ Gap-NMF can work well on analyzing and processing
recorder music, it can be applicable to other types of audio.
■Thank you!