Outline
• Classification – continued
• Parameter estimation
Assumptions
• Suppose that there are c categories
– {1, 2, ....., c}
• The prior probability and class conditional
density are known
• There are a possible actions
– {1,  2, .....,  a}
• Loss function (i | j} describe the loss
incurred for taking action i when the state of
nature is j
5/29/2016
Visual Perception Modeling
2
Loss Function
• The expected loss function given a particular
observation x
c
R( i | x)    ( i |  j ) P( j | x)
j 1
• The overall risk
R   R( ( x) | x) p( x)dx
5/29/2016
Visual Perception Modeling
3
Bayes Decision Rule
• To minimize the overall risk, compute the
conditional risk and select the action for
which the conditional risk is minimum
c
R( i | x)    ( i |  j ) P( j | x)
j 1
– The resulting minimum overall risk is called the
Bayes risk, which is the best performance
5/29/2016
Visual Perception Modeling
4
Minimum-Error-Rate Classification
• Zero-one loss
• For minimum error rate,
– Decide 1 if P(1 | x) > P(2 | x)
– This is the Bayes decision rule
5/29/2016
Visual Perception Modeling
5
Discriminant Functions
• The classifier is said to assign a feature
vector x to class i if
– gi(x) > gj(x) for all j  i
– This can be viewed as a network
– If f(.) is a monotonically increasing function,
f(g(x)) and g(x) as discriminant function will
give the same classification result
5/29/2016
Visual Perception Modeling
6
Decision Regions
• The effect of decision rule is to divide the
feature space into c decision regions
– R1, R2, ...., Rc
– The regions are separated by decision boundaries
5/29/2016
Visual Perception Modeling
7
Normal Density
• Gaussian density
p ( x) 
1
(2 ) d / 2 | Σ |1/ 2
 1

t 1
exp  ( x μ )  ( x μ )
 2

– Properties
•
•
•
•
5/29/2016
Mean
Variance
Entropy
Central limit theorem
Visual Perception Modeling
8
Discriminant Functions for Normal Density
• Minimum error rate classification for normal
density
1
d
t 1
g i ( x)   ( x   i )  i ( x   i )  ln( 2 )
2
2
1
 ln(|  i |)  ln( P( i ))
2
5/29/2016
Visual Perception Modeling
9
Normal Density – Cont.
• Case I:
– i =  2 I
– Linear discriminant function
– Minimum distance classifier as a special case
• Template matching
5/29/2016
Visual Perception Modeling
10
Normal Density – Cont.
• Case II
– i = 
– Mahalanobis distance
( x  i )  ( x  i )
T
1
– The resulting discriminant function is also linear
5/29/2016
Visual Perception Modeling
11
Normal Density – Cont.
• Case III
– i arbitrary
– Hyper-quadric
5/29/2016
Visual Perception Modeling
12
Bayes Decision Theory – Discrete Features
• In this case, a feature can only take one of the
m discrete values v1, ....., vm
• To minimize the overall risk, select the action
that minimizes
  arg min R(i | x)
*
i
5/29/2016
Visual Perception Modeling
13
Parameter Estimation
• We could design an optimal classifier if we
knew the prior probabilities and the classconditional densities
– Unfortunately, in pattern recognition applications we
rarely have this kind of complete knowledge about
the probabilistic structure of the problem
• Training data
– Some vague, general knowledge about the problem
– A number of design samples
5/29/2016
Visual Perception Modeling
14
Parameter Estimation – cont.
• Two approaches
– Parameter estimation
• Estimate the parameters of the unknown probabilities
and probability densities
– Non-parametric procedures
• Multi-layer perceptrons and in general neural
networks
• Fisher linear discriminant function
5/29/2016
Visual Perception Modeling
15
Parameter Estimation – cont.
• Parameter estimation
– Maximum-likelihood approach
• Parameters as quantities whose values are fixed but
unknown
• The best estimate of their value is the one that
maximizes the probability of obtaining the samples
– Bayesian learning
• Parameters are random variables with known prior
distribution
• Observations convert the prior into posteriori
5/29/2016
Visual Perception Modeling
16
Maximum-Likelihood Estimation
• The general principle
– Log-likelihood
• Gaussian cases
– Unknown 
– Unknown  and 
5/29/2016
Visual Perception Modeling
17
Bayesian Estimation
• Class-conditional densities
• Parameter Distribution
• Gaussian case
– Univariate case
– Multivariate case
5/29/2016
Visual Perception Modeling
18
Bayesian Estimation – cont.
• General theory
5/29/2016
Visual Perception Modeling
19