NEURAL NETWORKS
Introduction
Hazırlayan
PROF. DR. YUSUF OYSAL
NEURAL NETWORKS – Introduction
INTRODUCTION TO NEURAL NETWORKS
• Introduction
• Single-Layer Perceptron Networks
• Learning Rules for Single-Layer Perceptron Networks
– Perceptron Learning Rule
– Adaline Leaning Rule
– -Leaning Rule
• Multilayer Perceptron
• Back Propagation Learning algorithm
NEURAL NETWORKS – Introduction
REFERENCES
•
•
•
•
•
•
•
•
Picton, P. (2000). Neural Networks, 2nd Ed. Basingstoke, UK.: Palgrave.
Haykin, S. (1999). Neural Networks: A Comprehensive Foundation, 2nd Ed.
Upper Saddle River, NJ.: Prentice-Hall Inc.
Callan, R. (1999). The Essence of Neural Networks. Hemel Hempstead, UK.:
Prentice-Hall Europe.
Pinel, J.P.J. (2003). Biopsychology, 5th Ed. Boston, MA.: Allyn & Bacon.
Kohonen, T. (1997). Self-Organizing Maps, 2nd Ed. Berlin, Heidelberg, New
York: Springer-Verlag.
Bishop, C.M. (1995). Neural Networks for Pattern Recognition. Oxford, UK:
Clarendon Press.
Arbib, M.A. (Ed) (2003). The Handbook of Brain Theory and Neural Networks,
2nd Edition. Cambridge, MA.: MIT Press.
Beale, R. & Jackson, T. (1990). Neural Computing: An Introduction. Bristol, UK.:
Institute of Physics Publishing.
NEURAL NETWORKS – Introduction
Historical Background
• 1943 McCulloch and Pitts proposed the first computational
models of neuron.
• 1949 Hebb proposed the first learning rule.
• 1958 Rosenblatt’s work in perceptrons.
• 1969 Minsky and Papert’s exposed limitation of the theory.
• 1970s Decade of dormancy for neural networks.
• 1980-90s Neural network return (self-organization, backpropagation algorithms, etc)
• 1990-Today Advanced applications based fast learning
algorithms
NEURAL NETWORKS – Introduction
Nervous Systems
• UNITs: nerve cells called neurons, many different types and
are extremely complex
• Human brain contains ~ 1011 neurons.
• INTERACTIONs: signal is conveyed by action potentials,
interactions could be chemical (release or receive
neurotransmitters) or electrical at the synapse. Each neuron is
connected ~ 104 others.
• Some scientists compared the brain with a “complex, nonlinear,
parallel computer”.
• The largest modern neural networks achieve the complexity
comparable to a nervous system of a fly.
NEURAL NETWORKS – Introduction
Neurons
NEURAL NETWORKS – Introduction
A Model of Artificial Neuron
x1
wi1
x2
m
f ( i ).   wij x j
.
.
yi (t  1)  a( f )
wi2
j 1
xm= 1

yi
f (.) a (.)
wim =i
bias
1 f  0
a( f )  
0 otherwise
NEURAL NETWORKS – Introduction
A Model of Artificial Neuron
• Graph representation:
– nodes: neurons
– arrows: signal flow
directions
y1
y2
yn
. . .
. . .
• A neural network that does
not contain cycles (feedback
loops) is called a feed–
forward network (or
perceptron).
. . .
. . .
x1
x2
xm
NEURAL NETWORKS – Introduction
Layered Structure
y1
y2
Output Layer
yn
. . .
. . .
Hidden Layer(s)
. . .
Input Layer
. . .
x1
x2
xm
NEURAL NETWORKS – Introduction
Knowledge and Memory
• The output behavior of a network is
determined by the weights.
• Weights  the memory of an NN.
• Knowledge  distributed across the
network.
• Large number of nodes
y1
y2
. . .
. . .
– increases the storage “capacity”;
– ensures that the knowledge is robust;
– fault tolerance.
• Store new information by changing
weights.
yn
. . .
. . .
x1
x2
xm
NEURAL NETWORKS – Introduction
Neuron Models
x0 = +1
x1
Input
signal
w1
x2

w2

xm
w0
Local
Field
v
Summing
function

wm Synaptic
weights
Activation
function
 ()
Output
y
Neuron Models
Sign function
Step function
Linear function
Y
Y
Y
Y
+1
+1
+1
+1
0
-1
Y
Sigmoid function
step
X
0
-1
X
0
X
-1
 1, if X  0 sigmoid
1, if X  0
1
sign
Y


Y

 1, if X  0
0, if X  0
1  e X
0
-1
Y linear  X
X
NEURAL NETWORKS – Introduction
Network architectures
 Three different classes of network architectures
 single-layer feed-forward neurons are organized
 multi-layer feed-forward
in acyclic layers
 recurrent
 The architecture of a neural network is linked with the
learning algorithm used to train
NEURAL NETWORKS – Introduction
Single Layer Feed-forward
Input layer
of
source nodes
Output layer
of
neurons
NEURAL NETWORKS – Introduction
Multi-layer feed-forward
3-4-2 Network
Output
layer
Input
layer
Hidden Layer(s)
NEURAL NETWORKS – Introduction
Recurrent network
z-1
z-1
z-1
input
hidden
output
NEURAL NETWORKS – Introduction
ANN Configuration
A net with unidirection
lateral connection
A net with feedback