Neural Networks
Le Song
Machine Learning I
CSE 6740, Fall 2013
Learning nonlinear decision boundary
Linearly separable
Nonlinearly separable
The XOR gate
Speech recognition
2
A decision tree for Tax Fraud
Input: a vector of attributes
𝑋 = [Refund,MarSt,TaxInc]
Output:
𝑌= Cheating or Not
H as a procedure:
Each internal node: test one
attribute 𝑋𝑖
Each branch from a node: selects
one value for 𝑋𝑖
Each leaf node: predict 𝑌
Refund
Yes
No
NO
MarSt
Single, Divorced
TaxInc
< 80K
NO
Married
NO
> 80K
YES
3
Apply model to test data I
Query Data
Start from the root of tree.
R e fu n d
No
Refund
Yes
M a r it a l
T a x a b le
S ta tu s
In c o m e
C heat
M a r r ie d
80K
?
10
No
NO
MarSt
Single, Divorced
TaxInc
< 80K
NO
Married
NO
> 80K
YES
4
Apply model to test data II
Query Data
R e fu n d
No
Refund
Yes
M a r it a l
T a x a b le
S ta tu s
In c o m e
C heat
M a r r ie d
80K
?
10
No
NO
MarSt
Single, Divorced
TaxInc
< 80K
NO
Married
NO
> 80K
YES
5
Apply model to test data III
Query Data
R e fu n d
No
Refund
Yes
M a r it a l
T a x a b le
S ta tu s
In c o m e
C heat
M a r r ie d
80K
?
10
No
NO
MarSt
Single, Divorced
TaxInc
< 80K
NO
Married
NO
> 80K
YES
6
Apply model to test data IV
Query Data
R e fu n d
No
Refund
Yes
M a r it a l
T a x a b le
S ta tu s
In c o m e
C heat
M a r r ie d
80K
?
10
No
NO
MarSt
Single, Divorced
TaxInc
< 80K
NO
Married
NO
> 80K
YES
7
Apply model to test data V
Query Data
R e fu n d
No
Refund
Yes
M a r it a l
T a x a b le
S ta tu s
In c o m e
C heat
M a r r ie d
80K
?
10
No
NO
MarSt
Single, Divorced
TaxInc
< 80K
NO
Married
Assign Cheat to “No”
NO
> 80K
YES
8
Expressiveness of decision tree
Decision trees can express any function of the input attributes.
E.g., for Boolean functions, truth table row → path to leaf:
Trivially, there is a consistent decision tree for any training set
with one path to leaf for each example.
Prefer to find more compact decision trees
9
Decision tree learning
Tid
Attrib1
Attrib2
Attrib3
Class
1
Yes
Large
125K
No
2
No
Medium
100K
No
3
No
Small
70K
No
4
Yes
Medium
120K
No
5
No
Large
95K
Yes
6
No
Medium
60K
No
7
Yes
Large
220K
No
8
No
Small
85K
Yes
9
No
Medium
75K
No
10
No
Small
90K
Yes
Tree
Induction
algorithm
Induction
Learn
Model
Model
10
Training Set
Tid
Attrib1
11
No
Small
Attrib2
55K
Attrib3
?
12
Yes
Medium
80K
?
13
Yes
Large
110K
?
14
No
Small
95K
?
15
No
Large
67K
?
Apply
Model
Class
Decision
Tree
Deduction
10
Test Set
10
Top-Down Induction of Decision tree
Main loop:
𝐴 ← the “best” decision attribute for next node
Assign A as the decision attribute for node
For each value of A, create new descendant of node
Sort training examples to leaf nodes
If training examples perfectly classified, then STOP;
ELSE iterate over new leaf nodes
11
Tree Induction
Greedy strategy.
Split the records based on an attribute test that optimizes certain
criterion.
Issues
Determine how to split the records
How to specify the attribute test condition?
How to determine the best split?
Determine when to stop splitting
12
Example of a decision tree
T id
R e fu n d
Splitting Attributes
M a r it a l
T a x a b le
S ta tu s
In c o m e
C heat
1
Yes
S in g le
125K
No
2
No
M a r r ie d
100K
No
3
No
S in g le
70K
No
4
Yes
M a r r ie d
120K
No
5
No
D iv o r c e d
95K
Yes
6
No
M a r r ie d
60K
No
7
Yes
D iv o r c e d
220K
No
8
No
S in g le
85K
Yes
9
No
M a r r ie d
75K
No
10
No
S in g le
90K
Yes
Refund
Yes
No
NO
MarSt
Single, Divorced
TaxInc
< 80K
NO
Married
NO
> 80K
YES
10
Training Data
Model: Decision Tree
13
Another example of a decision tree
MarSt
T id
R e fu n d
Married
M a r it a l
T a x a b le
S ta tu s
In c o m e
C heat
NO
1
Yes
S in g le
125K
No
2
No
M a r r ie d
100K
No
3
No
S in g le
70K
No
4
Yes
M a r r ie d
120K
No
5
No
D iv o r c e d
95K
Yes
6
No
M a r r ie d
60K
No
7
Yes
D iv o r c e d
220K
No
8
No
S in g le
85K
Yes
9
No
M a r r ie d
75K
No
10
No
S in g le
90K
Yes
Single,
Divorced
Refund
No
Yes
NO
TaxInc
< 80K
NO
> 80K
YES
There could be more than one tree that
fits the same data!
10
Training Data
14
How to determine the Best Split
Idea: a good attribute splits the examples into subsets that are
(ideally) "all positive" or "all negative"
Homogeneous,
Non-homogeneous,
Low degree of impurity
High degree of impurity
Greedy approach:
Splitting nodes with homogeneous class distribution are preferred
Need a measure of node impurity
How to compare attribute?
Entropy
Entropy H(X) of a random variable X
Entropy quatifies the randomness of a random variable.
Information theory: H(X) is the expected number of bits needed to encode a
randomly drawn value of X (under most efficient code)
Examples for computing Entropy
C1
0
P(C1) = 0/6 = 0
C2
6
Entropy = – 0 log 0 – 1 log 1 = – 0 – 0 = 0
C1
1
P(C1) = 1/6
C2
5
Entropy = – (1/6) log2 (1/6) – (5/6) log2 (1/6) = 0.65
C1
2
P(C1) = 2/6
C2
4
Entropy = – (2/6) log2 (2/6) – (4/6) log2 (4/6) = 0.92
P(C2) = 6/6 = 1
P(C2) = 5/6
P(C2) = 4/6
Sample Entropy
S is a sample of training examples
p+ is the proportion of positive examples in S
p- is the proportion of negative examples in S
Entropy measure the impurity of S
How to compare attribute?
Conditional Entropy of variable 𝑋 given variable 𝑌
The entropy 𝐻 𝑋 𝑌 = 𝑗 of X given a specific value 𝑌 = 𝑗 :
Conditional entropy 𝐻 𝑋 𝑌 of X given Y: average of 𝐻 𝑋 𝑌 = 𝑗
𝑁
𝐻 𝑋𝑌 =−
𝑁
𝑁
𝑖
𝑗
𝑃 𝑦=𝑗 𝐻 𝑋𝑦=𝑗 =
𝑗
𝑃(𝑥 = 𝑖, 𝑦 = 𝑗)
𝑃 𝑥 = 𝑖, 𝑦 = 𝑗 log 2
𝑃(𝑦 = 𝑗)
Mutual information (aka information gain) of X given Y :
Information Gain
Information gain (after split a node):
ni

( p)   
Entropy
 i 1 n
k
GAIN
split
 Entropy

(i ) 

𝑛 samples in parent node 𝑝 is split into 𝑘 partitions; 𝑛𝑖 is number of
records in partition 𝑖
Measures Reduction in Entropy achieved because of the split.
Choose the split that achieves most reduction (maximizes
GAIN)
Stopping Criteria for Tree Induction
Stop expanding a node when all the records belong to the
same class
Stop expanding a node when all the records have similar
attribute values
Early termination
Biologically inspired learning model
Consider humans:
Neuron switching time
~ 0.001 second
Number of neurons
~ 1010
Connections per neuron
~ 104-5
Scene recognition time
~ 0.1 second
100 inference steps doesn't seem like enough
 much parallel computation
Perceptron
From biological neuron to
artificial neuron (perceptron)
In p u ts
x1
Synapse
De ndrites
Synapse
A xo n
L in e a r
w1
N o n lin e a r
C o m b in e r T r a n s fo r m
O u tp u t
A xo n

Soma
w2
Soma
De ndrites
Synapse

x2
T h r e s h o ld
I n p u t
O u t p u t
S i g n a l s
S i g n a l s
Artificial neural nets (ANN)
Many neuron-like threshold
switching units
Many weighted
interconnections among units
Highly parallel, distributed
processes
Y
M id d le L a y e r
In p u t L a y e r
O u tp u t L a y e r
23
Jargon Pseudo-Correspondence
Independent variable = input variable
Dependent variable = output variable
Coefficients = “weights”
Logistic Regression Model (the sigmoid unit) is a perceptron
Inputs
Age
34
Output
5
0.6
Gender
1
4
Stage
4
8
S
“Probability
of beingAlive”
Independent variables
Coefficients
Dependent variable
x1, x2, x3
a, b, c
p Prediction
What is logistic regression model
Assume that the posterior distribution 𝑝 𝑦 = 1 𝑥 take a
particular form
1
𝑝 𝑦 = 1 𝑥, 𝜃 =
1 + exp(−𝜃 ⊤ 𝑥)
Logistic function (or sigmoid function) 𝜎 𝑢 =
𝜕𝜎(𝑢)
𝜕𝑢
1
1+exp −𝑢
=𝜎 𝑢 1−𝜎 𝑢
25
Learning parameters in logistic regression
Find 𝜃, such that the conditional likelihood of the labels is
maximized
𝑚
𝑃 𝑦𝑖 𝑥𝑖 , 𝜃
max 𝑙 𝜃 : = log
𝜃
𝑖=1
Good news: 𝑙 𝜃 is concave function of 𝜃, and there is a single
global optimum.
Bad new: no closed form solution (resort to numerical method)
26
The objective function 𝑙 𝜃
logistic regression model
1
𝑝 𝑦 = 1 𝑥, 𝜃 =
1 + exp(−𝜃 ⊤ 𝑥)
Note that
1
exp −𝜃 ⊤ 𝑥
𝑝 𝑦 = 0 𝑥, 𝜃 = 1 −
=
⊤
1 + exp −𝜃 𝑥
1 + exp −𝜃 ⊤ 𝑥
Plug in
𝑚
𝑃 𝑦𝑖 𝑥𝑖 , 𝜃
𝑙 𝜃 : = log
𝑖=1
(𝑦 𝑖 − 1) 𝜃 ⊤ 𝑥 𝑖 − log(1 + exp −𝜃 ⊤ 𝑥 𝑖 )
=
𝑖
27
The gradient of 𝑙 𝜃
𝑚
𝑃 𝑦𝑖 𝑥𝑖 , 𝜃
𝑙 𝜃 : = log
𝑖=1
(𝑦 𝑖 − 1) 𝜃 ⊤ 𝑥 𝑖 − log(1 + exp −𝜃 ⊤ 𝑥 𝑖 )
=
𝑖
Gradient
𝜕𝑙(𝜃)
=
𝜕𝜃
𝑖
⊤𝑥 𝑖 𝑥 𝑖
exp
−𝜃
(𝑦 𝑖 − 1) 𝑥 𝑖 +
1 + exp −𝜃 ⊤ 𝑥
Setting it to 0 does not lead to closed form solution
28
Gradient Ascent algorithm
Initialize parameter 𝜃 0
Do
𝜃 𝑡+1 ← 𝜃 𝑡 + 𝜂
(𝑦 𝑖 − 1) 𝑥 𝑖 +
𝑖
𝑡⊤ 𝑖
exp −𝜃 𝑥 𝑥 𝑖
⊤
1 + exp −𝜃 𝑡 𝑥
While the ||𝜃 𝑡+1 − 𝜃 𝑡 || > 𝜖
29
Learning with square loss
Find 𝜃, such that the conditional probability of the labels are
close to the actual labels (0 or 1)
𝑛
𝑦𝑖
min 𝑙 𝜃 ≔
𝜃
−𝑃 𝑦 =1
𝑥𝑖, 𝜃
𝑖
2
𝑛
𝑦𝑖
=
−𝜎
𝜃⊤𝑥𝑖
2
𝑖
Not a convex objective function
Use gradient decent to find a local optimum
30
The gradient of 𝑙 𝜃
𝑛
𝑦𝑖
𝑙 𝜃 :=
−𝑃 𝑦 =1
𝑥𝑖, 𝜃
2
𝑛
𝑦𝑖
=
𝑖
−𝜎
𝜃⊤𝑥 𝑖
2
𝑖
Let 𝑢𝑖 = 𝜃 ⊤ 𝑥 𝑖
𝜕𝜎(𝑢)
𝜕𝑢
=𝜎 𝑢 1−𝜎 𝑢
Gradient
𝜕𝑙(𝜃)
=
𝜕𝜃
2 𝑦 𝑖 − 𝜎 𝑢𝑖
𝜎 𝑢𝑖
1 − 𝜎 𝑢𝑖
𝑥𝑖
𝑖
31
Gradient Descent algorithm
Initialize parameter 𝜃 0
Do
𝑢𝑖 : =
𝜃 𝑡+1 ← 𝜃 𝑡 − 𝜂
⊤ 𝑖
𝑡
𝜃 𝑥
2 𝑦 𝑖 − 𝜎 𝑢𝑖
𝜎 𝑢𝑖
1 − 𝜎 𝑢𝑖
𝑥𝑖
𝑖
While the ||𝜃 𝑡+1 − 𝜃 𝑡 || > 𝜖
32
Decision boundary a perceptron
x
y
Z (color)
0
0
1
0
1
1
1
0
1
1
1
0
NAND
w1
x1
y  = 0.5
w2
x2
f(x1 w1 + x2 w2 ) = y
f(0w1 + 0w2 ) = 1
f(0w1 + 1w2 ) = 1
f(1w1 + 0w2 ) = 1
f(1w1 + 1w2 ) = 0
some possible values for w1 and w2
f(a) =

w1
0.20
0.20
0.25
0.40
w2
0.35
0.40
0.30
0.20
1, for a > 
0, for a  
Difficult problem for perceptron
x
y
Z (color)
0
0
0
0
1
1
1
0
1
1
1
0
NAND
w1
x1
y  = 0.5
w2
x2
f(x1 w1 + x2 w2 ) = y
f(0w1 + 0w2 ) = 0
f(0w1 + 1w2 ) = 1
f(1w1 + 0w2 ) = 1
f(1w1 + 1w2 ) = 0
f(a) =

w1
some possible values for w1 and w2
w2
1, for a > 
0, for a  
Connecting perceptrons = neural networks
x
y
Z (color)
0
0
0
0
1
1
1
0
1
1
1
0
NAND
 = 0.5 for all units
w5
w6
f(a) =
w1 w w2
3
w4
a possible set of values for (w1,
(0.6,-0.6,-0.7,0.8,1,1)

1, for a > 
0, for a  
w2, w3, w4, w5 , w6):
Neural Network Model
Inputs
Age
.6
34
.2
.4
S
.5
.1
Gender
2
.2
.3
S
.7
Stage
4
Independent
variables
Output
.8
S
.2
Weights
Hidden
Layer
Weights
0.6
“Probability
of beingAlive”
Dependent
variable
Prediction
“Combined logistic models”
Inputs
Age
.6
34
.4
S
.5
.1
Gender
2
S
.2
.7
Stage
Output
S
“Probability
of beingAlive”
.8
4
Independent
variables
Weights
Hidden
Layer
0.6
Weights
Dependent
variable
Prediction
“Combined logistic models”
Inputs
Age
34
.4
.2
Gender
Stage
2
4
Independent
variables
Output
.5
S
S
.2
.3
S
“Probability
of beingAlive”
.8
.2
Weights
Hidden
Layer
0.6
Weights
Dependent
variable
Prediction
“Combined logistic models”
Inputs
Age
.6
34
.2
Output
.4
S
.5
.1
Gender
1
S
.3
S
.7
Stage
4
Independent
variables
“Probability
of beingAlive”
.8
.2
Weights
Hidden
Layer
0.6
Weights
Dependent
variable
Prediction
Not really, no target for hidden units...
Age
.6
34
.2
.4
S
.5
.1
Gender
2
.2
.3
S
.7
Stage
4
Independent
variables
.8
S
.2
Weights
Hidden
Layer
Weights
0.6
“Probability
of beingAlive”
Dependent
variable
Prediction
Training hidden units
Find 𝜃, 𝛼, 𝛽, such that the value of the output unit are close to
the actual labels (0 or 1)
𝑛
𝑦𝑖
min 𝑙 𝜃, 𝛼, 𝛽 : =
𝜃,𝛼,𝛽
where
−𝜎
𝜃 ⊤𝑧𝑖
2
𝑖
𝑧1𝑖 = 𝜎 𝛼 ⊤ 𝑥 𝑖
𝑧2𝑖 = 𝜎(𝛽⊤ 𝑥 𝑖 )
Not a convex objective function
Use gradient decent to find a local optimum
41
The gradient of 𝑙 𝜃, 𝛼, 𝛽
𝑙 𝜃, 𝛼, 𝛽 : =
𝑛
𝑖
𝑖
⊤ 𝑖
𝑦 −𝜎 𝜃 𝑧
2
where 𝑧1𝑖 = 𝜎 𝛼 ⊤ 𝑥 𝑖 , 𝑧2𝑖 = 𝜎(𝛽⊤ 𝑥 𝑖 )
Let 𝑢𝑖 = 𝜃 ⊤ 𝑧 𝑖
Gradient
𝜕𝑙(𝜃, 𝛼, 𝛽)
=
𝜕𝜃
2 𝑦 𝑖 − 𝜎 𝑢𝑖
𝜎 𝑢𝑖
1 − 𝜎 𝑢𝑖
𝑧𝑖
𝑖
𝑧 𝑖 is computed using 𝛼 and 𝛽 from previous iteration
𝑧1𝑖 = 𝜎 𝛼 ⊤ 𝑥 𝑖 , 𝑧2𝑖 = 𝜎(𝛽⊤ 𝑥 𝑖 )
42
Gradient for 𝛼 and 𝛽
Use chain rule of derivatives
Note 𝑧1𝑖 = 𝜎 𝛼 ⊤ 𝑥 𝑖 , 𝑧2𝑖 = 𝜎(𝛽⊤ 𝑥 𝑖 )
Let 𝑣 𝑖 = 𝛼 ⊤ 𝑥 𝑖
Gradient
𝜕𝑙(𝜃, 𝛼, 𝛽) 𝜕𝑙(𝜃, 𝛼, 𝛽) 𝜕𝑧1𝑖
=
𝜕𝛼
𝜕𝛼
𝜕𝑧1𝑖
2 𝑦 𝑖 − 𝜎 𝑢𝑖
=
𝜎 𝑢𝑖
1 − 𝜎 𝑢𝑖
𝜃1 𝜎 𝑣 𝑖
1 − 𝜎 𝑣𝑖
𝑥𝑖
𝑖
43
Expressive Capabilities of ANNs
Boolean functions:
Every Boolean function can be represented by network with
single hidden layer
But might require exponential (in number of inputs) hidden units
Continuous functions:
Every bounded continuous function can be approximated with
arbitrary small error, by network with one hidden layer [Cybenko
1989; Hornik et al 1989]
Any function can be approximated to arbitrary accuracy by a
network with two hidden layers [Cybenko 1988].
Abdominal Pain Perceptron
20
10
Ulcer
Pain
Cholecystitis
Duodenal Non-specific
Perforated
1
0
Appendicitis Diverticulitis
37
0
1
WBC
Obstruction Pancreatitis
Small Bowel
0
Temp
Age
0
Male
Intensity Duration
Pain
Pain
adjustable
1
1
weights
0
0
Myocardial Infarction Network
Duration
Pain
2
Intensity Elevation
Pain ECG: ST
4
1
Myocardial Infarction
0.8
Smoker
1
Age
50
Male
1
“Probability” of MI
The "Driver" Network
Application: ANN for Face Reco.
The model
The learned hidden unit weights