Outline
• Parameter estimation – continued
– Non-parametric methods
Maximum-Likelihood Estimation
• Assumptions
– We separate a collection of samples according to class
D1, D2, ....., Dc
– Samples in Dj are drawn independently according to
the probability p(x|wj)
– We assume that p(x|wj) has a known parametric form
and is uniquely determined by the value of a parameter
vector j
– To simplify further, we assume that samples in Di give
no information about j if i  j
5/29/2016
Visual Perception Modeling
2
Maximum-Likelihood Estimation – cont.
• Suppose that D contains n samples
– x1, ....., xn
– By assumption that samples were drawn
independently, we have
n
p ( D | θ )   p ( xk | θ )
k 1
– The maximum-likelihood estimate of  is the
value of * that maximizes p(D| )
5/29/2016
Visual Perception Modeling
3
Bayesian Estimation
• Assumptions
– The form of the density p(x|) is assumed to be
known, but the value of the parameter vector  is
not known exactly
– Our initial knowledge about  is assumed to be
contained in a known prior density p()
– The rest of our knowledge about  is contained in
a set D of n samples x1, ....., xn drawn
independently according to the unknown
probability density p(x)
5/29/2016
Visual Perception Modeling
4
Bayesian Estimation – cont.
• General theory
– The basic problem is to compute the posterior
density p(|D)
– By Bayes formula we have
p( | D) 
p( D |  ) p( )
 p( D |  ) p( )d
– By the independence assumption
n
p ( D |  )   p ( xk |  )
k 1
5/29/2016
Visual Perception Modeling
5
Bayesian Estimation – cont.
• Gaussian case
– The univariate case p(m|D)
5/29/2016
Visual Perception Modeling
6
Bayesian Estimation – cont.
• Gaussian case – continued
– The univariate case p(x|D)
– The multivariate case
5/29/2016
Visual Perception Modeling
7
Non-parametric Methods
• In maximum-likelihood and Bayesian
estimation
– The forms of the probability densities are
assumed to be known
– However, the assumed forms rarely fit the
densities in practice
• In particular, all of the classical parametric densities
are uni-modal
5/29/2016
Visual Perception Modeling
8
A Multimodal Density
5/29/2016
Visual Perception Modeling
9
Solutions
• More complicated parametric models
– Mixture of Gaussians
– More general, some basis functions to describe a
probability density
– Learning is intrinsically more difficult when we
have more parameters
• Non-parametric methods
5/29/2016
Visual Perception Modeling
10
Non-parametric Methods
• Most of the non-parametric density
estimation methods are based on the
following fact
– The probability P that a vector x will fall in a
region R is given by
P   p( x)dx
R
5/29/2016
Visual Perception Modeling
11
Non-parametric Methods – cont.
• For n smaples x1, ....., xn that are drawn
independently according to p(x), the
probability that k of n will be in R is given by
n k
n k


Pk    P (1  P)
[k]  nP
k 
P   p( x)dx  p( x)V
R
k/n
p( x) 
V
5/29/2016
Visual Perception Modeling
V is the volume of R
12
Non-parametric Methods – cont.
5/29/2016
Visual Perception Modeling
13
Non-parametric Methods – cont.
• Problems to be addressed
– If we fix the volume V and have more samples,
the ratio k/n will converge as desired
• Averaged version of p(x)
– How to estimate p(x)?
• Let V approach zero?
5/29/2016
Visual Perception Modeling
14
Parzen Windows
• Parzen windows
– We use a window function for interpolation, each
sample contributing to the estimate in accordance
with its distance from x
1 n 1  x  xi 

pn ( x)    
n i 1 Vn  hn 
• Here hn is a parameter
5/29/2016
Visual Perception Modeling
15
Parzen Windows - cont.
• Choice of hn
– Too large, the spatial resolution is low
– Too small, the estimate will have a large variance
5/29/2016
Visual Perception Modeling
16
Parzen Windows - cont.
• Properties
– Convergence of mean
• As n approaches infinity, the estimate will also
approach p(x) if p(x) is continuous
• Smaller Vn is better
– Convergence of variance
• A smaller variance needs a larger Vn
5/29/2016
Visual Perception Modeling
17
Parzen Windows - cont.
5/29/2016
Visual Perception Modeling
18
Parzen Windows - cont.
5/29/2016
Visual Perception Modeling
19
Parzen Windows - cont.
5/29/2016
Visual Perception Modeling
20
Parzen Windows - cont.
5/29/2016
Visual Perception Modeling
21
Kn-Nearest-Neighbor Estimation
• Let the cell volume be a function of the
training data
– To estimate p(x) from n samples, we can center a
cell about x and let it grow until it captures kn
samples
kn / n
pn ( x ) 
Vn
5/29/2016
Visual Perception Modeling
22
Kn-Nearest-Neighbor Estimation – cont.
5/29/2016
Visual Perception Modeling
23
Kn-Nearest-Neighbor Estimation – cont.
5/29/2016
Visual Perception Modeling
24
Kn-Nearest-Neighbor Estimation – cont.
5/29/2016
Visual Perception Modeling
25
The Nearest-Neighbor Rule
• The nearest-neighbor rule
– Let Dn={x1, ...., xn} denote a set of n labeled
prototypes
– Suppose that x' be the prototype nearest to a test
point x
– We classify x to the class associated with x'
5/29/2016
Visual Perception Modeling
26
The Nearest-Neighbor Rule – cont.
5/29/2016
Visual Perception Modeling
27