Outline
• Classification
Bayesian Decision Rule
• A two-class example
– 1 for sea bass
– 2 for salmon
• Prior probability
– P(1)
– P(2)
5/29/2016
Visual Perception Modeling
2
Bayesian Decision Rule – cont.
• Class conditional probability density
– P(1 | x)
– P(2 | x)
• Bayes formula
p( x |  i ) P( i )
P( i ) 
P( x)
5/29/2016
Visual Perception Modeling
3
Bayesian Decision Rule – cont.
• Bayes decision rule
– Decide 1 if P(1 | x) > P(2 | x)
– Otherwise decide 2
– The optimal decision rule
• Minimize the average error we make
5/29/2016
Visual Perception Modeling
4
Feature Space
• Feature space
– The Euclidean space Rd if we use a ddimensional feature
– Each possible feature is a point the Rd space
5/29/2016
Visual Perception Modeling
5
Loss Function
• Loss function
– States exactly how costly each action is
– Is used to convert a probability determination
into a decision
– Allows us to treat situations where some kinds of
classification mistakes are more costly than
others
• Equally costly is a special case
5/29/2016
Visual Perception Modeling
6
Loss Function – cont.
• Suppose that there are c categories
– {1, 2, ....., c}
• There are a possible actions
– {1,  2, .....,  c}
• Loss function (i | j} describe the loss
incurred for taking action i when the state of
nature is j
5/29/2016
Visual Perception Modeling
7
Loss Function – cont.
• Bayes formula
p( x |  i ) P( i )
P( i | x) 
P( x)
– where
c
p( x)   p( x |  j ) P( j )
j 1
5/29/2016
Visual Perception Modeling
8
Loss Function – cont.
• The expected loss function given a particular
observation x
c
R( i | x)    ( i |  j ) P( j | x)
• The overall risk
j 1
R   R( ( x) | x) p( x)dx
5/29/2016
Visual Perception Modeling
9
Bayes Decision Rule
• To minimize the overall risk, compute the
conditional risk and select the action for the
conditional risk is minimum
c
R( i | x)    ( i |  j ) P( j | x)
j 1
– The resulting minimum overall risk is called the
Bayes risk, which is the best performance
5/29/2016
Visual Perception Modeling
10
Two-Category Classification
• Two categories
• Two actions
• We decide 1 if
(21  11) p( x | 1 ) P(1 )  (12  22 ) p( x | 2 ) P(2 )
5/29/2016
Visual Perception Modeling
11
Minimum-Error-Rate Classification
• Zero-one loss
• For minimum error rate,
– Decide 1 if P(1 | x) > P(2 | x)
– This is the Bayes decision rule
5/29/2016
Visual Perception Modeling
12
Discriminant Functions
• The classifier is said to assign a feature
vector x to class i if
– gi(x) > gj(x) for all j  i
– This can be viewed as a network
– If f(.) is a monotonically increasing function,
f(g(x)) and g(x) as discriminant function will
give the same classification result
5/29/2016
Visual Perception Modeling
13
Decision Regions
• The effect of decision rule is to divide the
feature space into c decision regions
– R1, R2, ...., Rc
– The regions are separated by decision boundaries
– Two-category case
5/29/2016
Visual Perception Modeling
14
Normal Density
• Gaussian density
p ( x) 
1
(2 ) d / 2 |  |1/ 2
 1

t 1
exp  ( x   )  ( x   )
 2

– Properties
•
•
•
•
5/29/2016
Mean
Variance
Entropy
Central limit theorem
Visual Perception Modeling
15
Discriminant Functions for Normal Density
• Minimum error rate classification for normal
density
1
d
t 1
g i ( x)   ( x   i )  i ( x   i )  ln( 2 )
2
2
1
 ln(|  i |)  ln( P( i ))
2
– Special cases
5/29/2016
Visual Perception Modeling
16