Characterizing the Function Space
for Bayesian Kernel Models
Natesh S. Pillai, Qiang Wu, Feng Liang
Sayan Mukherjee and Robert L. Wolpert
JMLR 2007
Presented by: Mingyuan Zhou
Duke University
January 20, 2012
Outline
• Reproducing kernel Hilbert space (RKHS)
• Bayesian kernel model
– Gaussian processes
– Levy processes
• Gamma process
• Dirichlet process
• Stable process
– Computational and modeling considerations
• Posterior inference
• Discussion
RKHS
In functional analysis (a branch of mathematics),
a reproducing kernel Hilbert space is a Hilbert
space of functions in which pointwise evaluation
is a continuous linear functional. Equivalently,
they are spaces that can be defined
by reproducing kernels.
http://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space
A finite kernel based solution
The direct adoption of the finite representation is not a fully Bayesian model since it
depends on the (arbitrary) training data sample size . In addition, this prior distribution
is supported on a finite-dimensional subspace of the RKHS. Our coherent fully
Bayesian approach requires the specification of a prior distribution over the entire
space H.
Mercer kernel
Bayesian kernel model
Properties of the RKHS
Properties of the RKHS
Bayesian kernel models and integral operators
Two concrete examples
Two concrete examples
Bayesian kernel models
Gaussian processes
Levy processes
Levy processes
Poisson random fields
Poisson random fields
Dirichlet Process
Symmetric alpha-stable processes
Symmetric alpha-stable processes
Computational and modeling considerations
• Finite approximation for Gaussian processes
• Discretization for pure jump processes
Posterior inference
• Levy process model
– Transition probability proposal
– The MCMC algorithm
Classification of gene expression data
Classification of gene expression data
Discussion
• This paper formulates a coherent Bayesian perspective for
regression using a RHKS model.
• The paper stated an equivalence under certain conditions
of the function class G and the RKHS induced by the
kernel. This implies:
– (a) a theoretical foundation for the use of Gaussian processes, Dirichlet
processes, and other jump processes for non-parametric Bayesian kernel
models.
– (b) an equivalence between regularization approaches and the Bayesian
kernel approach.
– (c) an illustration of why placing a prior on the distribution is natural
approach in Bayesian non-parametric modelling.
• A better understanding of this interface may lead to a better
understanding of the following research problems:
–
–
–
–
Posterior consistency
Priors on function spaces
Comparison of process priors for modeling
Numerical stability and robust estimation