SEEM4630 2013-2014 Tutorial 2
Classification: Definition
 Given a collection of records (training set ), each record
contains a set of attributes, one of the attributes is the
class.
 Find a model for class attribute as a function of the
values of other attributes.
 Decision tree
 Naïve bayes
 k-NN
 Goal: previously unseen records should be assigned a
class as accurately as possible.
2
Decision Tree
 Goal
 Construct a tree so that instances belonging to

different classes should be separated
Basic algorithm (a greedy algorithm)
 Tree is constructed in a top-down recursive manner
 At start, all the training examples are at the root
 Test attributes are selected on the basis of a heuristics
or statistical measure (e.g., information gain)
 Examples are partitioned recursively based on selected
attributes
3
Attribute Selection Measure 1: Information Gain
 Let pi be the probability that a tuple belongs to class Ci,

estimated by |Ci,D|/|D|
Expected information (entropy) needed to classify a tuple
in D:
m
Info ( D )    p i log 2 ( p i )
i 1
 Information needed (after using A to split D into v
partitions) to classify D:
v
Info A ( D ) 

j 1
| Dj |
|D|
 Info ( D j )
 Information gained by branching on attribute A
Gain(A)  Info(D)  Info A (D)
4
Attribute Selection Measure 2: Gain Ratio
 Information gain measure is biased towards attributes
with a large number of values
 C4.5 (a successor of ID3) uses gain ratio to overcome the
problem (normalization to information gain)
v
SplitInfo
A
(D)  
| Dj |
j 1
|D|
 log 2 (
| Dj |
)
|D|
 GainRatio(A) = Gain(A)/SplitInfo(A)
5
Attribute Selection Measure 3: Gini index
 If a data set D contains examples from n classes, gini
index, gini(D) is defined as
n
gini ( D )  1   p 2j
j 1

where pj is the relative frequency of class j in D
If a data set D is split on A into two subsets D1 and D2,
the gini index gini(D) is defined as
gini A ( D ) 
| D1 |
|D |
gini ( D 1) 
|D 2 |
|D |
gini ( D 2 )
 Reduction in Impurity:  gini ( A )  gini ( D )  gini A ( D )
6
Example
Outlook
Temperature
Humidity
Wind
Play Tennis
Sunny
>25
High
Weak
No
Sunny
>25
High
Strong
No
Overcast
>25
High
Weak
Yes
Rain
15-25
High
Weak
Yes
Rain
<15
Normal
Weak
Yes
Rain
<15
Normal
Strong
No
Overcast
<15
Normal
Strong
Yes
Sunny
15-25
High
Weak
No
Sunny
<15
Normal
Weak
Yes
Rain
15-25
Normal
Weak
Yes
Sunny
15-25
Normal
Strong
Yes
Overcast
15-25
High
Strong
Yes
Overcast
>25
Normal
Weak
Yes
Rain
15-25
High
Strong
No
7
Tree induction example
 Entropy of data S
Info(S) = -9/14(log2(9/14))-5/14(log2(5/14)) = 0.94
 Split data by attribute Outlook
S[9+, 5-]
Outlook
Sunny [2+,3-]
Overcast [4+,0-]
Rain [3+,2-]
Gain(Outlook) = 0.94 – 5/14[-2/5(log2(2/5))-3/5(log2(3/5))]
– 4/14[-4/4(log2(4/4))-0/4(log2(0/4))]
– 5/14[-3/5(log2(3/5))-2/5(log2(2/5))]
= 0.94 – 0.69 = 0.25
8
Tree induction example
 Split data by attribute Temperature
S[9+, 5-]
Temperature
<15 [3+,1-]
15-25 [5+,1-]
>25 [2+,2-]
Gain(Temperature) = 0.94 – 4/14[-3/4(log2(3/4))-1/4(log2(1/4))]
– 6/14[-5/6(log2(5/6))-1/6(log2(1/6))]
– 4/14[-2/4(log2(2/4))-2/4(log2(2/4))]
= 0.94 – 0.80 = 0.14
9
Tree induction example
 Split data by attribute Humidity
High [3+,4-]
S[9+, 5-] Humidity
Normal [6+, 1-]
Gain(Humidity) = 0.94 – 7/14[-3/7(log2(3/7))-4/7(log2(4/7))]
– 7/14[-6/7(log2(6/7))-1/7(log2(1/7))]
= 0.94 – 0.79 = 0.15
 Split data by attribute Wind
Weak [6+, 2-]
S[9+, 5-] Wind
Strong [3+, 3-]
Gain(Wind) = 0.94 – 8/14[-6/8(log2(6/8))-2/8(log2(2/8))]
– 6/14[-3/6(log2(3/6))-3/6(log2(3/6))]
= 0.94 – 0.89 = 0.05
10
Tree induction example
Outlook
Temperat
ure
Humidity
Wind
Play
Tennis
Sunny
>25
High
Weak
No
Sunny
>25
High
Strong
No
Overcast
>25
High
Weak
Yes
Rain
15-25
High
Weak
Yes
Rain
<15
Normal
Weak
Yes
Rain
<15
Normal
Strong
No
Overcast
<15
Normal
Strong
Yes
Sunny
15-25
High
Weak
No
Sunny
<15
Normal
Weak
Yes
Rain
15-25
Normal
Weak
Yes
Sunny
15-25
Normal
Strong
Yes
Overcast
15-25
High
Strong
Yes
Overcast
>25
Normal
Weak
Yes
Rain
15-25
High
Strong
No
Gain(Outlook) = 0.25
Gain(Temperature)=0.14
Gain(Humidity) = 0.15
Gain(Wind) = 0.05
Outlook
Sunny
??
Overcast
Yes
Rain
??
11
 Entropy of branch Sunny
Info(Sunny) = -2/5(log2(2/5))-3/5(log2(3/5)) = 0.97
 Split Sunny branch by attribute Temperature
Sunny[2+,3-]
Temperature
<15 [1+,0-]
15-25 [1+,1-]
>25 [0+,2-]
Gain(Temperature)
= 0.97
– 1/5[-1/1(log2(1/1))-0/1(log2(0/1))]
– 2/5[-1/2(log2(1/2))-1/2(log2(1/2))]
– 2/5[-0/2(log2(0/2))-2/2(log2(2/2))]
= 0.97 – 0.4 = 0.57
 Split Sunny branch by attribute Humidity
Sunny[2+,3-]
Humidity
High [0+,3-]
Normal [2+, 0-]
Gain(Humidity)
= 0.97
– 3/5[-0/3(log2(0/3))-3/3(log2(3/3))]
– 2/5[-2/2(log2(2/2))-0/2(log2(0/2))]
= 0.97 – 0 = 0.97
 Split Sunny branch by attribute Wind
Sunny[2+, 3-]
Wind
Weak [1+, 2-]
Strong [1+, 1-]
Gain(Wind)
= 0.97
– 3/5[-1/3(log2(1/3))-2/3(log2(2/3))]
– 2/5[-1/2(log2(1/2))-1/2(log2(1/2))]
= 0.97 – 0.95= 0.02
12
Tree induction example
Outlook
Sunny
Humidity
High
No
Overcast
Yes
Rain
??
Normal
Yes
13
 Entropy of branch Rain
Info(Rain) = -3/5(log2(3/5))-2/5(log2(2/5)) = 0.97
 Split Rain branch by attribute Temperature
Rain[3+,2-]
Temperature
<15 [1+,1-]
15-25 [2+,1-]
>25 [0+,0-]
Gain(Outlook)
= 0.97
– 2/5[-1/2(log2(1/2))-1/2(log2(1/2))]
– 3/5[-2/3(log2(2/3))-1/3(log2(1/3))]
– 0/5[-0/0(log2(0/0))-0/0(log2(0/0))]
= 0.97 – 0.95 = 0.02
 Split Rain branch by attribute Humidity
Rain[3+,2-]
Humidity
High [1+,1-]
Normal [2+, 1-]
Gain(Humidity)
= 0.97
– 2/5[-1/2(log2(1/2))-1/2(log2(1/2))]
– 3/5[-2/3(log2(2/3))-1/3(log2(1/3))]
= 0.97 – 0.95 = 0.02
 Split Rain branch by attribute Wind
Rain[3+,2-]
Wind
Weak [3+, 0-]
Strong [0+, 2-]
Gain(Wind)
= 0.97
– 3/5[-3/3(log2(3/3))-0/3(log2(0/3))]
– 2/5[-0/2(log2(0/2))-2/2(log2(2/2))]
= 0.97 – 0 = 0.97
14
Outlook
Sunny
Humidity
High
No
Overcast
Rain
Yes
Normal
Yes
Wind
Weak
Yes
Strong
No
15
Bayesian Classification
 A statistical classifier: performs probabilistic prediction,
i.e., predicts class membership probabilities
 P ( C i | x1 , x 2 ,..., x n ) where xi is the value of attribute Ai
 Choose the class label that has the highest probability
 Foundation: Based on Bayes’ Theorem.
P ( C i | x1 , x 2 ,..., x n ) 
P ( x1 , x 2 ,..., x n | C i ) P ( C i )
P ( x1 , x 2 ,..., x n )

P ( C i | x1 , x 2 ,..., x n ) posteriori probability

P ( C i ) prior probability

P ( x1 , x 2 ,..., x n | C i ) likelihood
Model: compute
from data
16
Naïve Bayes Classifier
 Problem: joint probabilities are difficult to estimate
P ( x1 , x 2 ,..., x n | C i )
 Naïve Bayes Classifier
 Assumption: attributes are conditionally independent
P ( x1 , x 2 ,..., x n | C i )  P ( x1 | C i )  P ( x n | C i )
P ( C i | x1 , x 2 , ..., x n ) 

n
j 1
P ( x j | C i )P ( C i )
P ( x1 , x 2 , ..., x n )
17
Example: Naïve Bayes Classifier
A
B
C
m
b
t
m
s
t
g
q
t
h
s
t
g
q
t
g
q
f
g
s
f
h
b
F
h
q
f
m
b
f
P(C=t) = 1/2
P(C=f ) = 1/2
P(A=m|C=t) = 2/5
P(A=m|C=f ) = 1/5
P(B=q|C=t) = 2/5
P(B=q|C=f ) = 2/5
Test Record: A=m, B=q, C=?
18
Example: Naïve Bayes Classifier
For C = t
P(A=m|C=t) * P(B=q|C=t) * P(C=t) = 2/5 * 2/5 * 1/2
= 2/25 Higher!
P(C=t|A=m, B=q) = (2/25) / P(A=m, B=q)
For C = f
P(A=m|C=f) * P(B=q|C=f) * P(C=f) = 1/5 * 2/5 * 1/2
= 1/25
P(C=t|A=m, B=q) = (1/25) / P(A=m, B=q)
Conclusion: A=m, B=q, C=t
19
Nearest Neighbor Classification
Input
A set of stored records
k: # of nearest neighbors
Output
 Compute distance: d ( p , q )   ( p  q )
 Identify k nearest neighbors
 Determine the class label of unknown record based on class labels
2
i
i
i
of nearest neighbors (i.e. by taking majority vote)
20
Nearest Neighbor Classification
A Discrete Example
 Calculate the distances:
Input
Given 8 training instances
P1 (4, 2)  Orange
P2 (0.5, 2.5)  Orange
P3 (2.5, 2.5)  Orange
P4 (3, 3.5)  Orange
P5 (5.5, 3.5)  Orange
P6 (2, 4)  Black
P7 (4, 5)  Black
P8 (2.5, 5.5)  Black
d(P1, Pn) = ( 4  4 )
d(P2, Pn) = 3.80
d(P3, Pn) = 2.12
d(P4, Pn) = 1.12
d(P5, Pn) = 1.58
d(P6, Pn) = 2
d(P7, Pn) = 1
d(P8, Pn) = 2.12
2
 (2  4)  2
2
k=1&k=3
New Instance:
Pn (4, 4)  ?
21
Nearest Neighbor Classification
k=3
k=1
P8
P8
P7
P7
Pn
Pn
P6
P6
P2
P4
P3
P4
P5
P1
P2
P5
P3
P1
22
Nearest Neighbor Classification…
 Scaling issues
 Attributes may have to be scaled to prevent
distance measures from being dominated by one
of the attributes
• Each attribute must follow in the same range
• Min-Max normalization
 Example:
• Two data records: a = (1, 1000), b = (0.5, 1)
• dis(a, b) = ?
23
Classification: Lazy & Eager Learning
Two Types of Learning Methodologies
 Lazy Learning

• Instance-based learning. (k-NN)
Eager Learning
• Decision-tree and Bayesian classification.
• ANN & SVM
P8
P8
P7
P7
Pn
Pn
P6
P6
P4
P2
P4
P5
P3
P1
P2
P5
P3
P1
24
Differences Between Lazy &Eager Learning
 Lazy Learning
a. Do not require model building
b. Less time training but more time predicting
c. Lazy method effectively uses a richer hypothesis space
since it uses many local linear functions to form its
implicit global approximation to the target function
 Eager Learning
a. Require model building
b. More time training but less time predicting
c. Must commit to a single hypothesis that covers the
entire instance space
25
26