Teacher:
Elena Marchiori
R4.47
elena@cs.vu.nl
Assistant:
Kees Jong
S2.22 cjong@cs.vu.nl
Neural Networks NN 1 1

Basics of neural network theory and practice for supervised and unsupervised learning.
Most popular Neural Network models :
• architectures
• learning algorithms
• applications
Neural Networks NN 1 2

Rules: 4 s.p
- Final mark is based on two assignments , which will be available at the end of the course.
- one assignment is on theory (to do alone ).
- one assignment is on practice (to do in couples).
- Programming in Matlab 5.3.
- Registration: send email to cjong@cs.vu.nl
Neural Networks NN 1 3

• There is no text book.
• Course schedule, slides and exercises will be available at http://www.cs.vu.nl/~elena/nn.html
Neural Networks NN 1 4

• A NN is a machine learning approach inspired by the way in which the brain performs a particular learning task :
– Knowledge about the learning task is given in the form of examples .
– Inter neuron connection strengths ( weights ) are used to store the acquired information (the training examples).
– During the learning process the weights are modified in order to model the particular learning task correctly on the training examples .
Neural Networks NN 1 5

• Supervised Learning
– Recognizing hand-written digits, pattern recognition, regression.
– Labeled examples
(input , desired output)
– Neural Network models: perceptron , feed-forward , radial basis function , support vector machine .
• Unsupervised Learning
– Find similar groups of documents in the web, content addressable memory, clustering.
– Unlabeled examples
(different realizations of the input alone)
– Neural Network models: self organizing maps , Hopfield networks .
Neural Networks NN 1 6

• Three different classes of network architectures
– single-layer feed-forward neurons are organized
– multi-layer feed-forward in acyclic layers
– recurrent
• The architecture of a neural network is linked with the learning algorithm used to train
Neural Networks NN 1 7

Input layer of source nodes
Output layer of neurons
8 Neural Networks NN 1

3-4-2 Network
Output layer
Input layer
Neural Networks
Hidden Layer
NN 1 9

Recurrent Network with hidden neuron(s) : unit delay operator z -1 implies dynamic system z -1 z -1 input hidden output z -1
Neural Networks NN 1 10

Neural Network Architectures
Neural Networks NN 1 11

• The neuron is the basic information processing unit of a NN. It consists of:
1 A set of synapses or connecting links , each link characterized by a weight :
W
1
, W
2
, …, W m
2 An adder function (linear combiner) which the inputs: u
 j


1 w j x j
3 Activation function
 output of the neuron. y
 
(u
 b )
Neural Networks NN 1 12

Bias b x
1 w
1
Input signal x
2 w
2 x m
  w m
Synaptic weights
Neural Networks NN 1

Summing function
Local
Field v
Activation function

(

)
Output y
13

• Bias b has the effect of applying an affine transformation to u v = u + b
• v is the induced field of the neuron v u u
 j m


1 w j x j
14 Neural Networks NN 1

Input signal
•
Bias is an external parameter of the neuron . C an be modeled by adding an extra input.
w
0 v
 j m 

0 x
0
= +1 w j x j x
1 w
1 w
0
 b
Activation
Local function
Field x
2 w
2
 v

(

)
  Summing function x m
Neural Networks w m
Synaptic weights
NN 1 15
Output y

• Various types of neurons
• Various network architectures
• Various learning algorithms
• Various applications
Neural Networks NN 1 16

Face Recognition
90% accurate learning head pose, and recognizing 1-of-20 faces
Neural Networks NN 1 17

Neural Networks NN 1 18