Outline
• Parameter estimation – continued
– Bayesian learning
Announcement
• This coming Friday, March 2, there will be
no class during normal class time 10:1011AM.
• The class will be moved to 12:20-1:10PM in
Room 499, Dirac Science library
• Homework #2 is due today
5/29/2016
Visual Perception Modeling
2
Maximum-Likelihood Estimation
• Assumptions
– We separate a collection of samples according to class
D1, D2, ....., Dc
– Samples in Dj are drawn independently according to
the probability p(x|wj)
– We assume that p(x|wj) has a known parametric form
and is uniquely determined by the value of a parameter
vector j
– To simplify further, we assume that samples in Di give
no information about j if i  j
5/29/2016
Visual Perception Modeling
3
Maximum-Likelihood Estimation – cont.
• Suppose that D contains n samples
– x1, ....., xn
– By assumption that samples were drawn
independently, we have
n
p ( D | θ )   p ( xk | θ )
k 1
– The maximum-likelihood estimate of  is the
value of * that maximizes p(D| )
5/29/2016
Visual Perception Modeling
4
Maximum-Likelihood Estimation – cont.
• Log-likelihood
l (θ )  ln( p ( D | θ ))
θ*  arg max l (θ)
n
θ
l (θ)   ln( p ( xk | θ))
k 1
n
θ l (θ)    θ (ln( p ( xk | θ)) )
k 1
5/29/2016
Visual Perception Modeling
5
Maximum-Likelihood Estimation – cont.
• The maximum likelihood solution is
θl (θ)  0
– A solution * can be a true global maximum, a
local maximum, or a minimum, or an inflection
point of l()
• We need to check each solution individually
• Or calculate the second derivatives to identify the
global optimum
5/29/2016
Visual Perception Modeling
6
Bayesian Estimation
• We assume that parameters  are random
variables
– Training data allows us to convert a distribution
on the parameters into a posterior probability
density
– This is conceptually very different from
maximum-likelihood estimation
• There the parameters are fixed but not known
5/29/2016
Visual Perception Modeling
7
Bayesian Estimation – cont.
• Assumptions
– The form of the density p(x|) is assumed to be
known, but the value of the parameter vector  is
not known exactly
– Our initial knowledge about  is assumed to be
contained in a known prior density p()
– The rest of our knowledge about  is contained in
a set D of n samples x1, ....., xn drawn
independently according to the unknown
probability density p(x)
5/29/2016
Visual Perception Modeling
8
Bayesian Estimation – cont.
• General theory
– The basic problem is to compute the posterior
density p(|D)
– By Bayes formula we have
p( | D) 
p( D |  ) p( )
 p( D |  ) p( )d
– By the independence assumption
n
p ( D |  )   p ( xk |  )
k 1
5/29/2016
Visual Perception Modeling
9
Bayesian Estimation – cont.
5/29/2016
Visual Perception Modeling
10
Bayesian Estimation – cont.
• Recursive Bayesian learning
– Let Dn={x1, ....., xn}
– We have for n > 1
p( | D n ) 
p( xn |  ) p( | D n 1 )
n 1
p
(
x
|

)
p
(

|
D
)d
 n
– Incremental or on-line learning
• Learning goes on as the data are collected
5/29/2016
Visual Perception Modeling
11
Bayesian Estimation – cont.
5/29/2016
Visual Perception Modeling
12
Bayesian Estimation – cont.
• Bayesian learning
– As n increases, p(|Dn) becomes shaper and
shaper, approaching a Dirac delta function as n
goes to infinity. This behavior is commonly
known as Bayesian learning
• Relation to the maximum-likelihood solution
– If p(|Dn) is very peaky, then the solution from
the maximum-likelihood estimate would be close
to the true solution
5/29/2016
Visual Perception Modeling
13
Bayesian Estimation – cont.
• Why Bayesian estimation
– A measure of uncertainty about the current
estimation
5/29/2016
Visual Perception Modeling
14
Bayesian Estimation – cont.
• Why Bayesian estimation - continued
– Learning  recursively or incrementally
– Use all the available information to compute the
desired density p(x|D)
p( x | D)   p( x |  ) p( | D)d
5/29/2016
Visual Perception Modeling
15
Bayesian Estimation – cont.
• Gaussian case
– The univariate case p(m|D)
5/29/2016
Visual Perception Modeling
16
Bayesian Estimation – cont.
• Gaussian case – continued
– The univariate case p(x|D)
– The multivariate case
5/29/2016
Visual Perception Modeling
17