The artificial neuron revisited
The synthetic or artificial neuron, which is a simple model of the biological neuron,
was first proposed in 1943 by McCulloch and Pitts. It consists of a summing function
with an internal threshold, and "weighted" inputs as shown below.
For a neuron receiving n inputs, each input xi ( i ranging from 1 to n) is weighted by
multiplying it with a weight wi . The sum of the wixi products gives the net activation
of the neuron. This activation value is subjected to a transfer function to produce the
neuron’s output.
The weight value of the connection or link carrying signals from a neuron i to a
neuron j is termed wij..
wij
i
Direction of signal flow
j
Transfer functions
One of the design issues for ANNs is the type of transfer function used to compute the
output of a node from its net activation. Among the popular transfer functions are:
 Step function
 Signum function
 Sigmoid function
 Hyperbolic tangent function
In the step function, the neuron produces an output only when its net activation
reaches a minimum value – known as the threshold. For a binary neuron i, whose
output is a 0 or 1 value, the step function can be summarised as:
0 if activationi  T
outputi  
1 if activationi  T
When the threshold T is 0, the step function is called signum function.
533571934
1
Another type of signum function is
 1 if activationi  0

output i   0 if activationi  0
 1 if activation  0
i

The sigmoid transfer function produces a continuous value in the range 0 to 1. It has
the form:
outputi

1
1  e  gain . activationi
The parameter gain is determined by the system designer. It affects the slope of the
transfer around zero. The multilayer perceptron uses the sigmoid as the transfer
function.
A variant of the sigmoid transfer function is the hyperbolic tangent function. It has the
form:
outputi
e activationi  e  activationi

e activationi  e activationi
where u = gain . activationi. This function has a shape similar to the sigmoid (shaped
like an S), with the difference that the value of outputi ranges between –1 and 1.
533571934
2
output i
output i
Step function
Signum
1
1
0
0
T
activation i
activation i
output i
output i
1
1
Sigmoid
0.5
0
0
activation i
Figure Functional form of transfer functions
-1
activation i
Hyperbolic Tangent
Introduction to the multilayer perceptron
To be able to solve nonlinearly separable problems, a number of neurons are
connected in layers to build a multilayer perceptron.
Each of the perceptrons is used to identify small linearly separable sections of the
inputs.
Outputs of the perceptrons are combined into another perceptron to produce the final
output.
The hard-limiting (step) function used for producing the output prevents information
on the real inputs flowing on to inner neurons. To solve this problem, the step
function is replaced with a continuous function- usually the sigmoid function.
533571934
3
The Architecture of the Multilayer Perceptron
In a multilayer perceptron, the neurons are arranged into an input layer, an output
layer and one or more hidden layers.
The Generalised Delta Rule
The learning rule for the multilayer perceptron is known as "the generalised delta
rule" or the "backpropagation rule".
The generalised delta rule repetitively calculates an error function for each input and
backpropagates the error from one layer to the previous one.
The weights for a particular node are adjusted in direct proportion to the error in the
units to which it is connected.
Let
Ep
tpj
opj
wij
=
=
=
=
error function for pattern p
target output for pattern p on node j
actual output for pattern p on node j
weight from node i to node j
The error function Ep is defined to be proportional to the square of the difference tpj opj
(1)
Ep = 1/2(tpj - opj)2
j
The activation of each unit j, for pattern p, can be written as
netpj =  wijopi
i
533571934
(2)
4
The output from each unit j is determined by the non-linear transfer function fj
opj = fj(netpj)
We assume fj to be the sigmoid function, f(net) = 1/(1 + e-k.net),
where k is a positive constant that controls the "spread" of the function.
The delta rule implements weight changes that follow the path of steepest descent on
a surface in weight space. The height of any point on this surface is equal to the error
measure Ep. This can be shown by showing that the derivative of the error measure
with resepect to each weight is proportional to the weight change dictated by the delta
rule, with a negative constant of proportionality, i.e.,
pwi  -Ep/wij
The multilayer perceptron learning algorithm using the generalised delta
rule
1.
2.
3.
Initialise weights (to small random values) and transfer function
Present input
Adjust weights by starting from output layer and working backwards
wij(t + 1) = wij(t) + pjopi
wij(t) represents the weights from node i to node j at time t,  is a gain
term, and pj is an error term for pattern p on node j.
For output layer units
pj = kopj(1 - opj)(tpj - opj)
For hidden layer units
pj = kopj(1 - opj)  pkwjk
k
where the sum is over the k nodes in the following layer.
The learning rule in a multilayer perceptron is not guaranteed to produce
convergence, and it is possible for the network to fall into a situation (the so called
local minima) in which it is unable to learn the correct output.
533571934
5
Multilayer Perceptrons as Classifiers
The single layer perceptron is limited to calculating a single line of separation
between classes.
Let us consider a two layer perceptron with two units in the input layer.
If one unit is set to respond with a 1 if the input is above its decision line, and the
other responds with a 1 if the input is below its decision line, the second layer
produces a solution in the form of a 1 if its input is above line 1 and below line 2.
line 1
line 2
Fig. A 2-layer perceptron and the resluting decesion region.
A three layer perceptron can therefore produce arbitrarily shaped decision regions,
and are capable of separating any classes. This statement is referred to as the
Kolmogorov theorem.
Considering pattern recognition as a mapping function from unknown inputs to
known classes, any function, no matter how complex, can be represented by a
multilayer perceptron of no more than three layers.
The energy landscape
The behaviour of a neural network as it attempts to arrive at a solution can be
visualised in terms of the error or energy function Ep.
The energy is a function of the input and the weights. For a given pattern, Ep can be
plotted against the weights to give the so called energy surface. The energy surface is
a landscape of hills and valleys, with points of minimum energy corresponding to
wells and maximum energy found on peaks.
The generalised delta rule aims to minimise Ep by adjusting weights so that they
correspond to points of lowest energy. It does this by the method of gradient descent
where the changes are made in the steepest downward direction.
All possible solutions are depressions in the energy surface, known as basins of
attraction.
533571934
6
Learning Difficulties in Multilayer Perceptrons
Occasionally, the multilayer perceptron fails to settle into the global minimum of the
energy surface and instead find itself in one of the local minima. This is due to the
gradient descent strategy followed. A number of alternative approaches can be taken
to reduce this possibility:

Lowering the gain term  progressively

Addition of more nodes for better representation of patterns
Introduction of a momentum term  which determines the effect of past weight
changes on the current direction of the movement in weight space:

wij(t + 1) = wij(t) + 
pjopi + (wij(t) - wij(t - 1))
where momentum term 0 <  < 1.

Addition of random noise to perturb a system out of a local minima.
Advantages of Multilayer Perceptrons
The following two features characterise multilayer perceptrons and artificial neural
networks in general. They are mainly responsible for the "edge" these networks have
over conventional computing systems.
Generalisation
Neural networks are capable of generalisation, that is, they classify an unknown
pattern with other known patterns that share the same distinguishing features. This
means noisy or incomplete inputs will be classified because of their similarity with
pure and complete inputs.
Fault Tolerance
Neural networks are highly fault tolerant. This characteristic is also known as
"graceful degradation". Because of its distributed nature, a neural network keeps on
working even when a significant fraction of its neurons and interconnections fail.
Also, relearning after damage can be relatively quick.
533571934
7
Applications of Multilayer Perceptrons
The multilayer perceptron with backpropagation has been applied in numerous
applications ranging from OCR (Optical Character Recognition) to medicine. Brief
accounts of a few are given below.
Speech synthesis
A very well known use of the multilayer perceptron is NETtalk [1], a text-to-speech
conversion system, developed by Sejnowski and Rosenberg in 1987.
It consists of 203 input units, 120 hidden units, and 26 output units with over 27000
synapses. Each output unit represents one basic unit of sound, known as a phoneme.
Context is utilised in training by presenting seven successive letters to the input and
the net learns to pronounce the middle letter.
90% correct pronunciation achieved with the training set (80-87% with unseen set).
Resistant to damage and displays graceful degradation.
Multilayer perceptrons are also being used for speech recognition to be used in voice
activated control systems.
Financial applications
Examples include bond rating, loan application evaluation and stock market
prediction.
Bond rating involves categorising the bond issuer's capability. There is no hard and
fast rules for determining these ratings. Statistical regression is inappropriate because
the factors to be used are not well defined. Neural networks trained with
backpropagation has consistently outperformed standard statistical techniques [2].
Pattern Recognition
533571934
8
For many of the applications of neural networks, the underlying principle is that of
pattern recognition.
Target identification from sonar echoes has been developed. Given only a day of
training, the net produced 100% correct identification of the target, compared to 93%
scored by a Bayesian classifier.
There are many commercial applications of networks in character recognition. One
such system performs signature verification on bank cheques.
Networks have been applied to the problems of aircraft identification, and to terrain
matching for automatic navigation.
Limitations of Multilayer Perceptrons
1.
Computationally expensive learning process
Large number of iterations required for learning, not suitable for real-time
learning
2. No guaranteed solution
Remedies such as the "momentum term" add to computational cost
Other remedies:
using estimates of transfer functions
using transfer functions with easy to compute derivatives
using estimates of error values, eg., a single global error value for the
hidden layer
3. Scaling problem
Do not scale up well from small research systems to larger real systems.
Both too many and too few units slow down learning.
Biological arguments against Backpropagation

Backpropagation not used or used through different pathways in
biological systems.

Biological systems use only local information for self-adjustments.
The question one might ask at this point is - does an effective system need to
mimic nature exactly?
533571934
9
REFERENCES
Beale, R., & Jackson, T., "Neural Computing: An Introduction",
Bristol : Hilger, c1990.
(Contains full derivation of the generalised delta rule. Available at Murdoch
library)
533571934
10