2.160 System Identification, Estimation, and Learning
Lecture Notes No. 13
March 22, 2006
8. Neural Networks
8.1 Physiological Background
Neuro-physiology
• A Human brain has approximately 14 billion neurons, 50 different kinds of
neurons. … uniform
• Massively-parallel, distributed processing
Very different from a computer (a Turing machine)
Image removed due to copyright reasons.
McCulloch and Pitts, Neuron Model 1943
Donald Hebb, Hebbian Rule, 1949
…Synapse reinforcement learning
Rosenflatt, 1959
…The perceptron convergence theorem
Synapses
x1
The Hebbian Rule
Input xi
out put y
fired
fired
w1
x2
w2
xi
∑
wi
y
f
The i-th synapse wi is
reinforced.
wn
xn
1
The electric conductivity increases at wi
y
ŷ
z
g( z)
g
1
e
z
logistic function, or
sinusoid function
(1)
n
z = ∑ wi x i
Error e = yˆ − y = g ( z ) − y
i =1
Gradient Descent method
(2)
∆wi = − ρ ⋅ grad wi e2 = − ρ 2e
ρ : learning rate
(3) ∴ ∆wi = −2 ρ g ' e xi
Rule
∆wi ∝ (Input xi ).(Error)
Supervised Learning
∂e
∂wi
1
1 + e −z
Unsupervised Learning.
Replacing e by ŷ yields the Hebbian
(4) g ( z ) =
∆wi ∝ (Input xi ).(output ŷ )
8.2 Stochastic Approximation
consider a linear output function for yˆ = g ( z ) :
n
(5)
yˆ = ∑ wi xi
i =1
Given N sample data
{( y , x ,...x )
j
j
1
j = 1,...N
j
n
} Training Data
Find w1 ,..., wn that minimize
(6)
(7)
JN =
1
N
n
∑ ( yˆ
j
− y j)
i =1
2
∆wi = − ρ ⋅ grad wi J N = − ρ
N
∂yˆ j
( yˆ − y )
∑
∂wi
j =1
N
j
j
∂yˆ j
for all the sample date j=1,…N:
This method requires to store the gradient ( yˆ − y )
∂wi
It is a type of batch processing.
j
j
2
A simpler method is to execute updating the weight ∆wi every time the training data is
presented.
(8)
(9)
∆wi [k ] = ρδ [k ]xi [k ]
for the k-th presentation
Where δ (k ) = y[k ] − ∑ wl [k ]xl [k ]
Correct output for the
training data presented
at the k -th time
Predicted output based on the
weights wi [k ] for the training data
presented at the k -th time
Learning procedure
Present all the N training data in any sequence, and repeat the N presentations, called an
“epoch”, many times… Recycling.
N predictions
( x [1] , y [1]) ... ( x [ N ] , y [ N ]) .
epoch 1
epoch 2
epoch 3
epoch 4
epoch p
This procedure is called the Widrow-Hoff algorithm.
Convergence: As the recycling is repeated infinite times, does the weight vector
converge to the optimal one: arg min J N ( w1 ...w n ) ?... Consistency
w
If a constant learning rate ρ > 0 is used, this does not converge, unless min J N = 0
JN
Learning
Rate
ρk
ex. ρ k =
constant
k
w2
w1
W-space
W0
Minimum
point
0
k
3
constant
, convergence can be guaranteed.
k
This is a special case of the Method of Stochastic Approximation.
If the learning rate is varied, eg. ρ k =
Expected Loss Function
E [L( w)] = ∫ L( x, y w) p( x )dx
(10)
(11)
∫ L( x, y w) = 2 ( y − yˆ (x w))
1
2
The stochastic approximation procedure for minimizing this expected loss function with
respect to weight w1 ,..., wn is
∂
(12) wi [k + 1] = wi [k ] − ρ [k ]
L( x[k ], y [k ] w[k ])
∂wi
Where x[k ]is the training data presented at the k -th iteration. This estimate is proved
consistent if the learning rate ρ [k ] satisfies.
1) lim ρ [k ] = 0
k →∞
(13)
2).
k
lim ∑ ρ [i ] = +∞
k →∞
i =1
k
2
lim ∑ ρ [i ] < ∞
k →∞
(14)
i =1
This condition prevents all the
weights from converging so fast
that error will remain forever
uncorrected.
This condition ensures that random
fluctuations are eventually
suppressed
lim E[( wi [k ] − wi 0 ) 2 ] = 0 ;
k →∞
The estimated weights converge to their optimal values with probability of 1.
Robbins and Monroe, 1951
This stochastic Approximation method in general needs more presentation of data, i.e. the
convergence process is slower than the batch processing. But the computation is very
simple.
8.3 Multi-Layer Perceptrons
The Exclusive OR Problem
Input
0
0
1
1
X1
0
1
0
1
X2
Output
0
1
1
0
y
Can a single neural unit (perceptron)
with weights w1 , w2 , w3 , produce the
XOR truth table?
No, it cannot
4
x2
x1
g( z)
w1
x2
w2
1
∑
Class 1
g
z
wi
x1
1
Class 0
(15)
(16)
z = w1 x1 + w2 x 2 + w3
Set z=0, then 0 = w1 x1 + w2 x2 + w3 represents a straight line in the x1 − x2 plane.
⎧1
g( z) = ⎨
⎩0
z>0
z≤0
Class 0 and class 1 cannot be
separated by a straight line. …
Not linearly separable.
Consider a nonlinear function in lieu of (15)
1
(17)
z = f ( x1 , x 2 ) = x1 + x 2 − 2 x1 x 2 −
3
1
f (0,0) = −
3
Class 0
1
f (1,1) = −
3
f (1,0) = f (0,1) =
2
>0
3
Class 1
x2
z>0
z<0
Next, replace x1 x 2 by a new variable x3
1
(18)
z = x1 + x 2 − 2 x3 −
3
x1
This is apparently a linear function: Linearly Separable.
5
Hidden Units
• Augment the original input patterns
• Decode the input and generate tan internal representation
x1
z=0
x3
x2
z<0
+1
∑
-1.5
+1
-2
+1
+1
1
z>0
y
∑
-1/3
x2
x1
Hidden Unit
Not directly visible from output
Extending this argument, we arrive at a multi-layer network having multiple hidden
layers between input and output layers.
Multi-Layer Perception
Layer 2
Layer 1
Layer m
Layer M
Layer 0
Input
Layer
Hidden Layers
Output
Layer
6