Markov Chain Sampling
Methods for Dirichlet Process
Mixture Models
R.M. Neal
Summarized by Joon Shik Kim
12.03.15.(Thu)
Computational Models of
Intelligence
Abstract
• This article reviews Markov chain methods
for sampling from the posterior distribution
of a Dirichlet process mixture model and
presents two new classes of methods.
• One new approach is to make MetropolisHastings updates of the indicators
specifying which mixture component is
associated with each observation, perhaps
supplemented with a partial form of Gibbs
sampling.
Chinese Restaurant Process (1/2)
Chinese Restaurant Process (2/2)
Introduction (1/2)
• Modeling a distribution as a mixture of simpler
distribution is useful both as a nonparametric
density estimation method and as a way of
identifying latent classes that can explain the
dependencies observed between variables.
• Use of Dirichlet process mixture models has
become computationally feasible with the
development of Markov chain methods for
sampling from the posterior distribution of the
parameters of the component distribution
and/or of the associations of mixture
components with observations.
Introduction (2/2)
• In this article, I present two new
approaches to Markov chain sampling.
• A very simple method for handling nonconjugate priors is to use MetropolisHastings updates with the conditional prior
as the proposal distribution.
• A variation of this method may sometimes
sample more efficiently, particularly when
combined with a partial form of Gibbs
sampling.
Dirichlet Process Mixture Models
(1/5)
• The basic model applies to data y1,…,yn
which we regard as part of an indefinite
exchangeable sequence, or equivalent, as
being independently drawn from some
unknown distribution.
Dirichlet Process Mixture Models
(2/5)
• We model the distribution from which
the yi are drawn as a mixture of
distributions of the form F(θ), with the
mixing distribution over θ given G. We
let the prior for this mixing distribution
be a Dirichlet process, with concentration
parameter α and base distribution G0.
Dirichlet Process Mixture Models
(3/5)
Dirichlet Process Mixture Models
(4/5)
Dirichlet Process Mixture Models
(5/5)
• If we let K go to infinity, the conditional
probabilities reach the following limits:
Gibbs Sampling when Conjugate
Priors are used (3/4)
Nested CRP
Day 1
Day 2
Day 3