BAYESIAN INFERENCE
Sampling techniques
Andreas Steingötter
Motivation & Background
Exact inference is intractable, so we have to resort
to some form of approximation
Motivation & Background
• variational Bayes
– deterministic approximation not exact in principle
Alternative approximation:
• Perform inference by numerical sampling, also
known as Monte Carlo techniques.
Motivation & Background
Posterior distribution 𝑝 𝑧 is required (primarily) for the
purpose of evaluating expectations 𝐸(𝑓).
E( f ) 

f ( z ) p ( z )d ( z )
• 𝑓 𝑧 are predictions made by model with parameters 𝑧
• 𝑝 𝑧 is parameter prior and 𝑓 𝑧 = 𝑝(𝑦|𝑧) is likelihood evaluate the marginal likelihood (evidence) for a model
Motivation & Background
E( f ) 

f ( z ) p ( z )d ( z )
approximation
fˆ 
1
L

L
f (z
(l )
)
Classical Monte Carlo approx
l 1
𝑧 (𝑙) are random (not necessarily independent) draws from
𝑝 𝑧 , which converges to the right answer in the limit of large
numbers of samples, 𝐿.
Motivation & Background
fˆ 
1
L

L
l 1
f (z
(l )
)
if 𝑧 (𝑙) are independent draws from 𝑝 𝑧 , then low
numbers suffice to estimate expectation
Problems:
1. How to obtain independent samples from 𝒑 𝒛 ?
2. Expectation may be dominated by regions of small
probability -> large sample sizes will be required to
achieve sufficient accuracy
3. Monte Carlo ignores values of 𝑧 (𝑙) when forming
the estimate
How to do sampling?
1. Basic Sampling algorithms
– Restricted mainly to 1- / 2- dimensional problems
2. Markov chain Monte Carlo
– Very general and powerful framework
Basic sampling
Special cases
1. Model with directed graph
– Ancestral sampling:
•
p(z) 
M

p ( z i | pa i )
i 1
Easy sampling of joint distribution:
– Logic sampling:
•
Compare sampled value for 𝑧𝑖 with observed value at
node i. If NOT agree, then discard all previous samples
and start with first node
Random sampling
• Computers can generate only pseudorandom
numbers
– Correlation of successive values
– Lack of uniformity of distribution
– Poor dimensional distribution of output sequence
– Distance between where certain values occur are
distributed differently from those in a random
sequence distribution
Random sampling from the
Uniform Distribution
• Assumption:
– good pseudo-random generator for uniformly
distributed data is implemented
• Alternative: http://www.random.org
– “true” random numbers with randomness coming
from atmospheric noise
Random sampling from a standard
non-uniform distribution
Goal: Sample from non-uniform distribution 𝑝 𝑦
which is a standard distribution, i.e.
given in analytical form
Suppose: we have uniformly distributed random
numbers from (0,1)
Solution: Transform random numbers 𝑧 over (0,1)
using a function which is the inverse of the
indefinite integral of the desired distribution
Random sampling from a standard nonuniform distribution
• Step 1: Calculatey cumulative distribution function
z  h (y) 

p ( x )d ( x )

• Step 2: Transform samples 𝑈 𝑧 0,1 by
y  h(z)
1
Rejection sampling
Suppose:
– direct sampling from 𝑝 𝒛 is difficult, but
– 𝑝 𝒛 can be evaluated for any given value of 𝒛 up to
some normalization constant 𝑍
p(z) 
1
Z
p(z)
p
– 𝑍𝑝 is unknown, 𝑝 𝑧 can be evaluated
Approach:
– Define simple proposal distribution 𝑞(𝑧) such that
𝑘𝑞 𝑧 ≥ 𝑝(𝑧) for all 𝑧.
Rejection sampling
• Simple visual example
𝑘𝑞 𝑧
𝑝(𝑧)
• Constant k should be as small as possible.
• Fraction of rejected points depends on the ratio of the
area under the unnormalized distribution 𝑝 𝑧 to the
area under the curve 𝑘𝑞 𝑧 .
Rejection sampling
• Rejection sampler
– Generate two random numbers
1. number 𝑧0 from proposal distribution 𝑞(𝑧)
2. generate a number 𝑢0 from uniform distribution
over [0, k𝑞 𝑧0 ]
– If 𝑢0 > 𝑝 (𝑧0 ) reject!
– Remaining pairs have
unifrom distribution under
𝑝 (𝑧)
Adaptive rejection sampling
Suppose: difficult to determine a suitable analytic form
for the proposal distribution 𝑞(𝑧)
Approach: construct envelope function “on the fly”
based on observed values of the distribution 𝑝 𝑧
– if 𝑝 𝑧 is log concave (ln𝑝 𝑧 has non-increasing
derivatives) use derivatives to construct envelope
Adaptive rejection sampling
• Step 1: at initial set of grid points 𝑧1 , …, 𝑧𝑀 evaluate
function ln𝑝 𝑧𝑖 and its gradient and calculate
tangents at 𝑝 𝑧𝑖 , i = 1, … , M.
• Step 2: sample from envelop distribution, if accepted
use it to calculate 𝑝(𝑧), otherwise refine grid.
 Envelope distribution is a piecewise exponential
distribution q ( z )  k i  i exp{   i (z  z i 1 )}
Slope 
Offset k
z i 1  z  z i
Adaptive rejection sampling
Problem of rejection sampling:
• Find a proposal distribution 𝑞(𝑧), which is close to
required distribution to minimize rejection rate.
• Therefore restricted mainly to univariate
distributions curse of dimensionality
• However: potential subroutine
Importance sampling
• Framework for approximating expectations
𝐸𝑝 (𝑓 𝑧 ) directly with respect to 𝑝 𝒛
• Does NOT provide 𝑝 𝒛
E( f ) 

f ( z ) p ( z )d ( z )
fˆ 
1
L

L
f (z
(l )
)
l 1
Suppose (again):
– direct sampling from 𝑝 𝒛 is difficult, but
– 𝑝 𝒛 can be evaluated for any given value of 𝒛 up
to some normalization constant 𝑍.
Importance sampling
• As for rejection sampling, apply proposal distribution
𝑞 𝑧 from which it is easy to draw samples
Importance sampling
• Expectation formula for un-normalized distributions with
importance weights 𝑟𝑙
L
E( f )
w
l 1
l
f (z
(l )
)
wl 
rl

L
r
m 1 m

p(z

L
m 1
(l )
p(z
) / q(z
(m )
(l )
)
) / q(z
(m )
Key points:
– Importance weights correct bias introduced by sampling from
proposal distribution
– Dependence on how well 𝑞 𝑧 approximates 𝑝(𝑧) (similar to
rejection sampling)
• Choose sample points in input space where 𝑓 𝑧 𝑝 𝑧 is large (or at
least where 𝑝 𝑧 is large)
• If 𝑝 𝑧 > 0 in same region, then 𝑞 𝑧 > 0 necessary
)
Importance sampling
Attention:
• Consider none of the samples falls in the regions where
𝑓 𝑧 𝑝 𝑧 is large.
• In that case, the apparent variances of 𝑟𝑙 and 𝑟𝑙 𝑓(𝑧(𝑙)) may
be small even though the estimate of the expectation may be
severely wrong.
• Hence a major drawback of the importance sampling method
is the potential to produce results that are arbitrarily in error
and with no diagnostic indication.
 𝒒 𝒛 should NOT be small where 𝒑 𝒛 may be significant!!!
Markov Chain Monte Carlo (MCMC)
sampling
• MCMC is a general framework, sampling from large
class of distributions, scales well with dimensionality
of sample space
Goal: Generate samples from distribution 𝑝(𝑧)
Idea: Build a machine which uses the current sample
to decide which next sample to produce in such a way
that the overall distribution of the samples will be 𝑝(𝑧).
Markov Chain Monte Carlo (MCMC)
sampling
Approach:
1. Generate a candidate sample 𝑧 ∗ from a proposal distribution
𝑞(𝑧|𝑧 (𝜏) ) that depends on the current state 𝑧 (𝜏) and is
sufficiently simple to draw samples from directly.
2. Current sample 𝑧 (𝜏) is known (i.e. maintain record of the
current state)
3. Samples 𝑧 (1) , 𝑧 (2) , 𝑧 (3) , … form a Markov chain
4. Accept or reject the candidate sample 𝑧 ∗ according to some
appropriate criterion
z
(   1)
 z *

( )
 z
if accepted
if rejected
MCMC - Metropolis algorithm
Suppose:
– 𝑝 𝒛 can be evaluated for any given value of 𝒛 up
to some normalization constant 𝑍.
p(z) 
1
Z
p(z)
p
Algorithm:
• Step 1: Choose symmetric proposal distribution 𝑞 𝒛𝐴 𝒛𝐵 =
𝑞 𝒛𝐵 𝒛𝐴
• Step 2: Candidate sample 𝑧 ∗ is accepted with probability
MCMC - Metropolis algorithm
Algorithm (cont.):
• Step 2.1: Choose a random number 𝑢 with uniform
distribution in (0,1)
• Step 2.2: Acceptance test for 𝑢 <
• Step 3:
z
(  1)
*

z


( )
z


if accepted
if rejected
Metropolis algorithm
Notes:
• rejection of a points leads to the previous
sample (different from rejection sampling)
• If 𝑞 𝒛𝐴 𝒛𝐵 > 0 for any values 𝒛𝐴, 𝒛𝐵, then 𝒛(𝜏)
tends to 𝑝 𝒛 for 𝜏 -> 
• 𝑧 (1) , 𝑧 (2) , 𝑧 (2) ... present no independent
samples from 𝑝 𝒛 - serial correlation. Instead
retain only every Mth sample.
Examples: Metropolis algorithm
Implementation in R:
• Elliptical distibution
𝑝(𝑧 (𝜏) )
𝑝(𝑧 ∗)
𝑢<
Update state 𝒛
𝜏+1
Keep old state 𝒛
= 𝒛∗
𝜏+1
= 𝒛(𝜏)
Examples: Metropolis algorithm
Implementation in R:
• Initialization [-2,2], step size = 0.3
n=1500
n=15000
Examples: Metropolis algorithm
Implementation in R:
• Initialization [-2,2], step size = 0.5
n=1500
n=15000
Examples: Metropolis algorithm
Implementation in R:
• Initialization [-2,2], step size = 1
n=1500
n=15000
Validation of MCMC
Properties of Markov chains:
z(1)
p(z
z(2)
( m  1)
|z
(1)
, ... z
z(m)
(m )
)  p (z
(m  1)
|z
z(m+1)
(m )
)  T m (z
(m )
Transition probabilities 𝑇𝑚(𝑧 ′ , 𝑧):
If 𝑇𝑚 is the same for all m
homogeneous
Invariant
(stationary)
p (z) 
*
 T (z
z'
(m )
,z
( m  1)
*
) p (z
(m )
)
,z
(m  1)
)
Validation of MCMC
Propoerties of Markov chains: 𝑝∗ 𝑧 :
If 𝑇𝑚 is the same for all m
homogeneous
Invariant
(stationary)
p (z) 
*
 T (z
(m )
,z
( m  1)
*
) p (z
(m )
)
z'
Sufficient
detailed
balance
𝑇𝑚 satisfy p * ( z )T (z, z')  p * ( z ')T ( z ', z )
reversible
Validation of MCMC
Goal: invariant Markov chain that converges to
desired distribution 𝑝∗ 𝑧
𝑝∗ 𝑧 = lim 𝑝 𝑧m
invariant
𝑚→∞
!!! for any 𝑝(𝑧0)
ergodicity
• An ergodic Markov chain has only one equilibrium
distribution
Properties and validation of MCMC
Approach: Construct appropriate transition
probabilities 𝑇(𝑧 ′ , 𝑧):
• 𝑇(𝑧 ′ , 𝑧) from set of base transitions 𝑩k
– Mixture form
K
T ( z ', z ) 

k 1
– Successive application
k
B k ( z ', z )
k - Mixing coefficients
Metropolis-Hastings algorithm
• Generalization of Metropolis algorithm
– No symmetric proposal distribution 𝑞(𝑧) required
If symmetry
– Choice of proposal distribution crititcal
Metropolis-Hastings algorithm
• Gaussian centered on current state
• Small variance -> high acceptance, slow walk,
dependent samples
• Large variance -> high rejection rate
Gibbs sampling
• Special case of Metropolis-Hastings algorithm
– the random value is always accepted,
Suppose: 𝑝 𝑧1, 𝑧2, 𝑧3 ,
• Step 1: initial samples 𝑧1, 𝑧2, 𝑧3
• Step 2: (repeated)
–
–
–
𝑧11~𝑝 𝑧1 𝑧2, 𝑧3
𝑧21~𝑝(𝑧2|𝑧11, 𝑧3)
𝑧31~𝑝(𝑧3|𝑧11, 𝑧21)
1
• repeated by cycling
• randomly choose variable
to be updated
Gibbs sampling
• 𝑝 𝒛\i is invariant (unchanged)
• Univariate conditional distribution 𝑝(𝑧𝑖|𝒛\i) is
invariant (by definition)
 Joint distribution 𝑝 𝒛 is invariant
• Because
(fixed at each step)
Gibbs sampling
 Sufficient condition for ergodicity:
– None of the conditional distributions be
anywhere zero, i.e. any point in 𝒛 space can be
reached from any other point in a finite number of
steps
z(2)
z(1)
z(3)
Gibbs sampling
Obtain m independent samples:
1. Sample MCMC during a «burn-in» period to
remove dependence on initial values
2. Then, sample at set time points (e.g. every Mth
sample)
• The Gibbs sequence converges to a stationary
(equilibrium) distribution that is independent of
the starting values,
• By construction this stationary distribution is the
target distribution we are trying to simulate.
Gibbs sampling
• Practicability dependent feasibility to draw
samples from conditional distributions 𝑝(𝑧𝑖|𝒛\i).
• Directed graphs will lead to conditional
distributions for Gibbs sampling that are log
concave.
Adaptive rejection sampling methods