Outline
Latent Variable Models
Probabilistic PCA (PPCA)
Dual Probabilistic PCA (DPPCA)
Gaussian Latent Variable Models
MDS, Kernel PCA, Nonlinear variants
Unified objective function
Experiments
Extensions
Gaussian Process Dynamical Models
Hierarchical GPLVM
Motivation
To learn a low dimensional representation of
high dimensional data
Our primary focus will be on visualization of
the data in 2D
Desiderata:
Maintains proximity data (e.g. what’s close in
data space is close in latent space)
Probabilistic framework
Projection from latent space to data space
A non-linear latent space embedding
Notation
D = # of observed dimensions
q = # of latent dimensions
N = # of data points
Y = observed data
X = latent data
W = linear mapping
Probabilistic PCA
A prior is placed over X, our latent embedding space s.t X ~ N(0,I). X is then
marginalized out.
Marginalize X
Optimize W
Y is observed
Columns of U are the eigenvectors of S
L is a diagonal matrix of eigenvalues
R is an arbitrary orthogonal matrix
Taking the marginal likelihood of Y
by integrating over X results in PPCA.
Optimizing the log likelihood w.r.t W
results in solving the eigenvalue problem
for our sample covariance matrix S.
Generative Model of PPCA
Bishop et. al 2006
Z represents the latent space
Which then gets mapped
to the data space X via W
The green contours show
the marginal density for X
Dual Probabilistic PCA
A prior is placed over W, our mapping parameter s.t W ~ N(0, I). W is then
marginalized out.
Optimize X
Marginalize W
Y is observed
Columns of U are the eigenvectors of S
L is a diagonal matrix of eigenvalues
R is an arbitrary orthogonal matrix
Taking the marginal likelihood of Y
by integrating over X results in PPCA.
The marginal likelihood replaces W with X,
but the ML estimate takes a similar form…
DPPCA = PPCA
Solution for PPCA:
Solution for Dual PPCA:
Lawrence claims these are “equivalent” through the
relation:
Connection to Gaussian Processes
Recall that a Gaussian process is fully defined by its mean m(x)
and covariance function.
Assuming a 0 mean function
and a covariance function K
Recall that the marginal likelihoods for DPPCA takes this form:
This is equivalent to a product over D independent Gaussian processes
with the linear covariance function mentioned above.
Kernel PCA vs. GPLVM
Kernel PCA: Y X
Y
Y
Nonlinear
Embedding
GPLVM: X Y
W
Y
Linear
Mapping
via W
K’(Y)
W
X
Linear
Mapping
via W
X
X
K(X)
Nonlinear
Embedding
Unifying Objective
GPLVM can be connected to classical MDS and kernel
PCA via a unifying objective function.
Kernel PCA: S is non-linear & K is linear
GPLVM: K is non-linear & S is linear
Classical MDS: S is a proximity or similarity matrix
GPLVM can be optimized via SCG, scaled conjugate
gradient.
Sparsification
Kernel methods may be sped up through sparsification
In SCG optimization, each gradient computation requires
an inversion of the kernel matrix
This is
datasets
and thus prohibitive for visualizing large
We can represent the data with an Informative Vector
Machine
Through a subset I of d points known as the active set with
likelihood
This is
dominated by selection of the active set
(Sparse) Oil Flow Visualizations
Radial Basis Function Kernel
Leads to smooth functions that fall
away to zero in regions with no data.
Multi-layer Perceptron Kernel
Leads also to smooth functions,
but regions with no data tend to
have the same values.
Oil Flow Visualization
Full (No Sparsification) RBF Kernel
Generative Topographic Mapping
Alternative Noise Models
We are NOT constrained to Gaussian noise models
We visualize handwritten 2’s with two noise models
Gaussian Noise
Test
Pixels removed in red
Baseline Model
Gaussian Model
Bernoulli Model
Bernoulli Noise
Hallucinating Faces
These faces were created by taking 64 uniformly spaced and ordered points from the latent space (1D)
and visualizing the mean of their distribution in data space.
Examples from the data-set which are closest to the corresponding fantasy images in latent space.
Pros and Cons of GPLVM
Pros
Probabilistic
Missing data straightforward.
Can sample from model given X.
Different noise models can be handled.
Flexible family of kernels.
Cons
Speed of optimisation.
Optimisation is non-convex using a non-linear kernel.
Sensitivity to initializations.
Initialization Issues
GPLVM initialized with PCA
Swiss Roll Dataset
GPLVM initialized with Isomap
Gaussian Process Dynamical Models
Nonlinear
Dynamical System
Generative Process
GPDM
GPDM marginalizes
over parameters A and B
Jack M. Wang et. al, Gaussian Process Dynamical Models (NIPS 2006)
GPDM on walking data
Training Data of Walking Video
GPDM of walk data learned
with an RBF Kernel +
2nd order dynamics
Missing Data
Training data with frames missing from
51 to 100, out of 158 frames in total.
Missing frames recovered by GPDM
with linear + RBF kernel
Visualization of Latent Space
GPLVM
Random Samples from X
GPDM
Data Space Mappings
MORE GPDM Action
GPDM learned using a 2nd order RBF Kernel for dynamics
Hierarchical GPLVM
We want
MAP Estimates
of this
Neil D. Lawrence Hierarchical Gaussian Process Latent Variable Models (ICML
High Five!
Provides
coordination
information
between the
2 subjects
Frames
A: 85, B: 114, C:127, D: 141, E: 155, F: 170, G: 190, H: 215
A motion capture data set consisting of two interacting subjects. The data,
which was taken from the CMU MOCAP database, consists of two subjects
that approach each other and ‘high five’.
Run, don’t walk!
The skeleton is decomposed as shown in Figure 4. In the plots, crosses are latent positions associated with the run
and circles are associated with the walk. We have mapped three points from each motion through the hierarchy.
Periodic dynamics was used in the latent spaces.
Conclusion
0
