NEURAL NETWORKS
Doç.Dr. Sibel SENAN
ssenan@iuc.edu.tr
1
GENERAL INFORMATION
ABOUT THE COURSE
2
GRADING
 MidTerm Exam
 Final Exam
50%
50%
The exams will include material covered in lecture.
3
RECOMMENDED TEXTBOOKS
(references for the content of the lecture)
• Jacek M. Zurada, Introduction to Artificial Neural
Systems, PWS Publishing Company, 1995.
• Simon Haykin, Neural Networks: A Comprehensive
Foundation, Macmillan College Publishing Company,
1994.
• Mohamad H. Hassoun, Foundamentals of Artificial
Neural Networks, The MIT Press, 1995.
• Laurene Fausett, Fundamentals of Neural Networks:
Architectures, Algorithms, and Applications, Prentice
Hall International, Inc., 1994.
4
POTENTIAL COURSE PLAN











Introduction to Neural Networks
History of Neural Networks
Basic Concepts
Learning in Neural Networks
Learning Tasks in Neural Networks
Multilayer Neural Networks
Hopfield Neural Networks
Bidirectional Associative Memory Neural Networks
Cellular Neural Networks
Deep Learning
Convolutional Neural Networks
5
INTRODUCTION TO
NEURAL NETWORKS
WEEK 01
6
Artificial Neural Network

An artificial neural network (ANN),
usually called neural network (NN),
is a mathematical model or computational model that is
inspired by the structure and/or functional aspects of
biological neural networks.

In an Artificial Nural Network,
simple artificial nodes, variously called neurons,
are connected together to form a network of nodes
mimicking the biological neural networks.
7
Biological Nervous System
 Central to the system is the brain, represented by the neural net, which
continually receives information, perceives it, and makes appropriate
decisions.
 The arrows pointing from left to right indicate the forward transmission
of information-bearing signals through the system.
 The arrows pointing from right to left signify the presence of feedback in
the system.
 The receptors convert stimuli from the human body or the external
environment into electrical impulses that convey information to the brain
(neural net)
 The effectors convert electrical impulses generated by the neural net
into discernible responses as system outputs.
8
Biological Neuron
•
Dendrites are branches that receive information from other neurons.
•
Cell nucleus or Soma processes the information received from dendrites.
•
Axon is a cable that is used by neurons to send information.
•
Synapse is the connection between an axon and other neuron dendrites.
9
From Biological to Artificial Neuron
10
Artificial Neuron

Artificial neurons are the constitutive units in an
artificial neural network.
The artificial neuron receives one or more inputs
(representing the one or more dendrites)
 Inputs are weighted
(representing the synapses)

The weighted inputs are summed
(representing the soma)
 The sum is passed through a nonlinear
activation function to produce an output
(representing a biological neuron's axon).

11
A Model of ith neuron in an ANN
12
A single synaptic connection between two neurons
in an ANN.
13
 Main elements of Artificial Neuron :
1. Synaptic Weights
(the type and amount of energy)
2. Inputs
(the amount of incoming signals)
3. Adder (Net Input Function)
(the net input of neuron)
4. Activation Function
(the behaviour of neuron)
(implies nonlinearity to the system)
5. Output
(the amount of outcoming signal)
14
Nonlinear model of j. Neuron in an ANN
x1
wj1
Net input
function
x2
wj2
.
.
.
.
.
.
xn
wjn
inputs
Weight
coefficients
.
.
.
∑
Activation
function
uj
f (.)
Net
input
yj
output
bj
Bias
15

wj1 : the effect of 1st neuron to j. neuron
wj2 : the effect of 2nd neuron to j. neuron
.
.
wjn : the effect of n. neuron to j. neuron

∑ (Summing Function/Adder/Net Input Function) :
determines the total effect of input signals
uj : Net input of j. Neuron
n
u j   w ji .xi
i 1
16
Notations for j. Neuron in an ANN
n = Number of neurons in ANN
xi = Input Signal
x = [x1 x2 …. xn]T
wji = Weight coefficients (from i. neuron to j. neuron)
n
uj = Net Input of j. Neuron
u j   w ji .xi
i 1
f (.) = Activation Function
n
yj = f (uj+bj ) = Output
yj = f (  w ji xi  b j )
i 1
17
Artificial Neural Network
w11
x1
x2
xN
w12
1
w1N
2
N
u1
u2
uN
f1(.)
y1
f2(.)
y2
fN(.)
yN
 Feedforward
y = f(u) = f (W.x)
 Single Layer
18
Artificial Neural Network
 Feedforward
 Multilayer
19
Artificial Neural Network
yn+1 = f(yn, u)
 Feedback
(at least one feedback loop)
 Single Layer
20
The Architecture of ANN
 Number of inputs and outputs of the network;
 Number of layers;
 How the layers are connected to each other;
 The activation function of each layer;
 Number of neurons in each layer.
21
Learning in Neural Networks
By adjusting the weights of an artificial neuron
we can obtain the output we want for specific
inputs. But when we have an ANN of hundreds
or thousands of neurons, it would be quite
complicated to find by hand all the necessary
weights. But we can find algorithms which can
adjust the weights of the ANN in order to obtain
the desired output from the network.
 This process of adjusting the weights is called
learning or training.

22
Various Neural Network Models
 Hopfield Neural Networks
 Cellular Neural Networks
 Cohen-Grossberg Neural Networks
 Neutral Neural Networks
 Bidirectional Associative Memory Neural
Networks
 Convolutional Neural Networks
 Etc.
23
Applications of Neural Networks
 Optimization Problems
 Associative Memory
 Pattern Recognition i.e. recognizing
handwritten characters
 Image Processing
 Noise Removal
 Signal Processing
 Etc.
24
Potential Dynamical Behaviors of NNs
 Stability
 Chaos
 Oscillations
 Unstability
 Limit Cycles
 Periodic Solution
25

W = {wji} : Weight Matrix
n=3
0 3 1
W=
𝑥 = [1
0.5
0.2]𝑇
5 0 6
2 4 -1
n
u j   w ji .xi
i 1
𝑓 . = 𝑠𝑔𝑛(. )
𝑦 = [𝑦1 𝑦2 𝑦3 ]𝑇 =?
wj1 wj2 wj3 : (rows)
weights come into neuron j
w1i w2i w3i : (columns)
weights go out of neuron i
26
Solution in Scalar Form
𝑥𝑖 = [1
0.2]𝑇
0.5
n
u j   w ji xi
i 1
For neuron (1)
j=1
𝑢1 = 𝑤11 𝑥1 + 𝑤12 𝑥2 + 𝑤13 𝑥3 = 0.1 + 3.0,5 + 1.0,2 = 1,7
For neuron (2)
j=2
𝑢2 = 𝑤21 𝑥1 + 𝑤22 𝑥2 + 𝑤23 𝑥3 = 5.1 + 0.0,5 + 6.0,2 = 7,2
For neuron (3)
j=3
𝑢3 = 𝑤31 𝑥1 + 𝑤32 𝑥2 + 𝑤33 𝑥3 = 2.1 + 4.0,5 − 1.0,2 = 3,8
27
𝑦1 = 𝑓 𝑢1 = 𝑠𝑔𝑛 1,7 = 1
𝑦2 = 𝑓 𝑢2 = 𝑠𝑔𝑛 7,2 = 1
𝑦3 = 𝑓 𝑢3 = 𝑠𝑔𝑛 3,8 = 1
𝑦 = [1 1 1 ]𝑇
28
Solution in Vector-Matrix Form
𝑦 = 𝑓(𝑢)
𝑢1
𝑢2
𝑢3
=
𝑦1
𝑢1
𝑦2 = 𝑓 𝑢2
𝑦3
𝑢3
0 3 1
1
5 0 6
0,5
2 4 -1
0,2
𝑢 = [𝑢1 𝑢2 𝑢3 ]𝑇
𝑢 = 𝑊𝑥
0.1 + 3.0,5 + 1.0,2
=
5.1 + 0.0,5 + 6.0,2
2.1 + 4.0,5 − 1.0,2
1,7
=
7,2
3,8
29
𝑓(𝑢1 )
𝑠𝑔𝑛 1,7
𝑦 = 𝑓(𝑢2 ) = 𝑠𝑔𝑛 7,2
𝑓(𝑢3 )
𝑠𝑔𝑛(3,8)
𝑦 = [1 1 1 ]𝑇
30