Outline
• Announcement
• Neural networks
– Perceptrons - continued
– Multi-layer neural networks
• Back-propagation
– Applications

Announcement
• Talk by Julian Besag
– Department of Statistics, Florida State University
– 3:30pm, Friday, March 9, 2001
– Room 001 OSB
• Project presentation and report
– Presentation and demo will be in Week 15
• Two of you will share a class
– Report is due on 5:00pm, Wednesday, April 25,
2001
4/16/2020 Visual Perception Modeling 2

McCulloch and Pitts Model w i1 w in

4/16/2020 Visual Perception Modeling 3

Activation Functions
• Activation functions
– O = g(x)
4/16/2020 Visual Perception Modeling 4

Issues of Neural Networks
• Issues to be solved when using neural networks
– What kind of architecture one should use?
– How to determine the connection weights?
• The main advantage of using neural networks is that there exist efficient learning algorithms which can determine the connection weights automatically for a large class of neural networks
4/16/2020 Visual Perception Modeling 5

Layered Feed-Forward Networks
• Layered feed-forward networks were called perceptrons
4/16/2020 Visual Perception Modeling 6

Simple Perceptrons
• Simple perceptrons
– One-layer feed-forward network
• There is an input layer and an output layer and no hidden layers
– The computation can be described by
O i
 g ( h i
)
 g (
 k w ik
 k
)
• Thresholds are omitted because they can always be treated as connections to an input terminal that is –1 permanently
4/16/2020 Visual Perception Modeling 7

A Simple Learning Algorithm
• There is a learning algorithm for a simple perceptron network
– Given a training pattern  k

, the desired output is
 i

– The learning algorithm, or the procedure to change its weights, is new w ik
 w ik
 old w ik
  w ik
 
(
 i
 
O i

)
 k

4/16/2020 Visual Perception Modeling 8

Perceptron Classification Demo
• The feature space is the two-dimensional plane
• We have three training examples
– One from the black category
– Two from the white category
– The line represents the decision boundary
• The network has two input neurons and one output
4/16/2020 Visual Perception Modeling 9

Simple Perceptrons – cont.
• Convergence of the learning rule
– One can prove mathematically that the learning rule will converge to a solution in case that the solution exists in finite learning steps
4/16/2020 Visual Perception Modeling 10

Simple Perceptrons – cont.
• Linear separability
– For simple perceptrons, the condition for correct operation is that a plane should divide the inputs that have positive and negative targets
– This means the decision boundary will be a plane where w
 x
 w
 x


0 if x
 is a positive example

0 if x
 is a negative example
• The plane is w •

= 0
4/16/2020 Visual Perception Modeling 11

Simple Perceptrons – cont.
• Linear units
O i
 g ( h i
)
  k w ik
 k
• Gradient descent learning
E [ w ]

1
2 i ,


(
 i

 w ik

O i

)
2 
  

E
 w ik

1
2 i ,


(
 i
   k w ik
 k

)
2
 

(
 i
 
O i

)
4/16/2020 Visual Perception Modeling 12

Simple Perceptrons – cont.
• Limitations of linear feed-forward networks
– A multi-layer linear feed-forward network is exactly equivalent to a one-layer one in the computation it performs
• Linear transformations of a linear transformation is a linear transformation
– Historically, this is a very important fact
• All linear feed-forward networks cannot solve linearly non-separable problems
– XOR problem
4/16/2020 Visual Perception Modeling 13

Multi-layer Perceptrons
• The limitations of perceptrons do not apply to feed-forward networks with hidden layers between the input and output layer with nonlinear activation function
• The problem is to train the network efficiently
4/16/2020 Visual Perception Modeling 14

Multi-layer Perceptrons – cont.
4/16/2020 Visual Perception Modeling 15

Multi-layer Perceptrons – cont.
• Back-propagation
– Extension of the gradient descent learning rule
E [ w ]

1
2 i ,


(
 i
 
O i

)
2 
1
2 i ,


[
 i
  g (
 j w ij g (
 k w ik
 k

))]
2
– The hidden-to-output layer connections
 w ik
  


E w ik
  
 g

( h i

)(
 i
 
O i

) V j

4/16/2020 Visual Perception Modeling 16

Multi-layer Perceptrons – cont.
• Back propagation - continued
– Input-to-hidden connections
 w ik
  

E
 w jk
  

[ g

( h j

)
 i w ij
 i

(
 i
 
O i

) V j
 where
 i
  
 g

( h i

)(
 i
 
O i

)
4/16/2020 Visual Perception Modeling 17

Activation Function
• Activation function
– For back-propagation, the activation function must be differentiable
– Also we want it to saturate at both extremes
– Sigmoid function g ( h )

1

1 exp(

2
 h ) g

( h )

2
 g ( 1
 g )
4/16/2020 Visual Perception Modeling 18

Activation Function – cont.
4/16/2020 Visual Perception Modeling 19

Back Propagation Algorithm
1.
Initialize the weights to small random values
2.
Choose a pattern
 k u and apply it to the input layer
3.
Propagate the signal forward through the network
4.
Compute the deltas (errors) for the output layer
5.
Compute the deltas (errors) for the preceding layers by propagating the errors backwards
6.
Update all the connections according to the algorithm
7.
Go back to step 2 and repeat for the next pattern
4/16/2020 Visual Perception Modeling 20

Using Neural Networks
• Design phase
– The neural network architecture
• Training phase
– Use available examples to train the neural network
• That is, to use the back-propagation algorithm to learn the connection weights
• Test phase
– For a new sample, feed the feature through the neural network and you go the result
4/16/2020 Visual Perception Modeling 21

Applications
• Application examples
– NETtalk
– Navigation of a car
– Image compression
– Recognizing hand-written ZIP codes
– Speech recognition
– Face recognition
4/16/2020 Visual Perception Modeling 22