Algorithm:
Step0: initialize wts using either the Hebb rule or the Delta rule.
Step1: For each input vector do step 2-4.
Step2: Set activations for input layer units equal to the current vector
Xi = Si
Step3: Compute the net input to the output units
Y-mj = ∑ Xi Wi j
Step4:Determine the activation of the output units for bipolar targest
1
Yj = 0
-1
if y – mj > 0
if y- mj = 0
if y – mj < 0
The output vector y gives the pattern associated with the input vector x .The hetero
associative memory is not iterative.
If the responses of the net are binary , a suitable activation fn is given by
1
if X > 0
0
if X <= 0
F(x)=
An example :(Hetroassociatwe net using the Hebb rule )suppose the net is to be trained to store the
following mapping from input bipolar vectors S= (S1, S2, S3, S4)
to output vectors with two components t = (t1,t2)
where
S(1) = (1, -1, -1, -1)
t (1) = (1 , -1)
S(2) = (1, 1, -1, -1)
t(2) = (1, -1)
S(3) = (-1, -1, -1, 1)
t (3) = (-1, 1)
S(4) = (-1, -1, 1, 1)
t (4) = (-1, 1)
The weight matrix is
Wij =
Σρ Si (ρ) tj ( ρ)
Hence
4
-4
2
-2
-2
2
-4
4
W =
Where the weight matrix to store the first potties pais is given by the outer product of the
vectors s = (1, -1, -1, -1) and t = (1, -1) , the weight matrix is
1
1
-1
-1
1
-1
-1
1
-1
-1
1
-1
1
-1
=
Similarly to store the 2nd , 3rd , 4th pais.The weight matrix to store all four pattem
pairs is sum of the weight matrices to store each pattem pais separately, i.e.
1
-1
1
-1
1
-1
1
1
-1
+ 1
-1
1
-1
1
1
-1
1
-1
1
-1
+
-1
1
-1
-1 +
1
-1
-1
-1
1
-1
2
4
-4
2
-2
1
-2
2
1
-4
4
=
Testing the net using the training input , note that the net input to any particular output unit
is the dot product of the input vector (row) with the column of the weight matrix that has
the weights for the output unit is question . The row vector with all of the net puts is the
product of the input vector and the at matrix . i.e. the entire is represented as
xw=(y.in1 , y.in2 )
Hence using the fist input vector
1st
( 1, -1, -1, -1 ) .W = ( 8, -8 ) → ( 1, -1 )
2nd
( 1, -1, -1, -1 ) . W = ( 12,- 12 ) → ( 1, -1 )
3rd
( -1, -1,-1, 1 ) . W = ( -8, 8 ) → ( -1, 1 )
4th
( -1, -1, 1, 1 ) . W = ( -12, 12 ) → ( -1, 1 )
Using the activation fn. for bipolar targets i.e.
1 if x > 0
F(x) =
0 if x =0
-1 if x < 0
The above results show that the correct response has been obtained for all of the training
patterns.
However, using an input vector x = (-1, 1, -1, -1) which is similar to the training vector s =
(1, 1, -1, -1) (the 2nd training vector) and differs only in the first component
Hence (-1, 1, -1, -1) . W = (4, -4)
(1, -1)
i.e. the net associates a known output pattern with this input
Testing the net with the pattern x = (-1, 1, 1, -1) which differs from each of the training
patterns in at least two components
(-1, 1, 1, -1). W = (0, 0)
(0, 0)
i.e. the output is not one of the outputs with which the net was training i.e. the net does
not recognize the pattern.
3
Autoassociative Net
In this net the process of training is often called for storing the vectors, which may be
binary or bipolar . A stored vector can be retrieved from distorted or noisy input if the input is
sufficiently similar to it.
The performance of the net is judged by its ability to reproduce a stored pattern from
noisy input.
Often, the weights on the diagonal (wts that connect an input pattern component to the
corresponding component in the output pattern) are set to zero.
Setting these wts to zero may improve the net's ability to generalize(when more than one
vector is stored in it).
Usually the weights are set using Hebb rule, the
X1
y1
Xi
Yi
Xn
Yn
outer product i.e.
p
W = ∑ S t (p) S (p)
P=1
The auto associative net can be used to determine
Whether an input vector is "known" or "unknown"
(i.e. not stored in the net ) .The procedure is :
Step0: Set the weights using Hebb rule,outer product
Step1: For each testing input vector , do step 2-4
Step2: Set activations of the input units equal to the input vector
Step3: Compute the net in put to each output unit
yinj = Σ xi wij
i
Step 4: Apply activation fn.
Yj = f (y.inj) = 1 if y.inj > 0
= -1 if y.inj ≤ 0
4
Example:An auto autoassociative net to store one vector (1, 1, 1, -1)
Step 0: compute the weights for the vector (1, 1, 1, -1)
1 1 1 -1
1 1 1 -1
W =
1 1 1 -1
-1 -1 -1 1
Step 1: For testing the input vector (recognized the stored vector).
Step 2:
x = (1, 1, 1, -1)
Step 3:
y.in = (4, 4, 4, -4)
Step 4:
y = f (4, 4, 4, -4) = (1, 1, 1, -1)
i.e. The input vector is recognized.
N.B. Using the net is simply
( 1, 1, 1, -1, ) . W
Similarly it can be show that the net recognizes the Vectors ( -1, 1, 1, -1, )
( 1, -1, 1, -1, ) , ( 1, 1, -1, -1, ) and
( 1, 1, 1, -1, )
i.e. the net recognizer vector with one mistake in input vector.
when one component is missing i.e.
( 0, 1, 1, -1, ) , (1, 0, 1, -1 ) , ( 1, 1, 0, 1 ) , ∞ ( 1, 1, 1, 0 )
Again , the net can recognize input vectors with two
missing entrées much as : ( 0, 0, 1, -1 ) , ( 0, 1, 0, -1 ) ,
(0, 1, 1, 0 ) , (1, 0, 0, -1 ) , (1, 0, 1, 0 ) ∞ (1, 1, 0, 0 )
5
However , it can be shown that the net does not recognize input vector with two mistakes in
the input vector much as (-1, -1, 1, -1 )
in general , a net is more tolerant of missing data than it is of mistakes in the data .
it is fairly common for an autoassociative net to have its diagonal terms set to zero . e.g.
0 1 1 -1
W0 = 1 0 1 -1
1 1 0 -1
-1 -1 -1 0
Using this net , it still does not recognize as input vector .
Such as (-1,-1.1,-1) with two mustaches.
Storage capacity:
This is numbers of vectors (or pattern pairs) that can stored in the net be in the net
before the net begins to forget.
More than one vector can be stored in an autoassociative net by adding the weight
matrices for each vector together.
The capacity of an auto associative depends on the net of components the stored
vectors have and The relationships among the stored vectors: more vectors can be
stored if they are mutually orthogonal.
An auto associative net with four nodes can store three orthogonal vectors (to each
of the other two) However, the wt metrics for four mutually orthogonal vectors is
always singular ( all the elements are zero)
6
7
8
Iterative autoassociative net:
Can make the input &output units be the same to obtain a recusant autoassociative
net i.e
X1.w =x2 iterate
X2.w = x3 and so on.
9
Discrete Hopfield net:
This net is fully interconnected (i.e. each unit is connected to every other unit .The
net has symmetric weight with no self connections, that is
Wij = Wji
and Wii=0
In the Hopfield net
1-only one unit updated its activation at a time (based on the signal it receives form
each other unit.
2 each unit continues to receive an external signal in addition to the signal form the
other units in the net.
10
11
12
13
N.B.:
There are variations on the discrete Hopfield net e.g. some use binary or bipolar
activations, others do not use external input after the first time.
14
The Hopfield net can be used to determine whether an input vector is a "known" (or
stored vector )or an "unknown" vector. The net recognizer a known vector by
producing a pattern of activation on the unites of the net that is the name an thestored
vector . if the input
vector is an unknown vector , the activation vectors produced as the net iterates will
converge to an activation vector that is not one of the stored vectors.
Example:
Stability Energy Function
W.Cohen and Grossbeng (1983) have shows that recumbent networks are stable if
the matrix is symmetrical with zeros in its main diagonal i.e. if Wij = Wji for i ≠ j &
Wii =0 for all i
Considering an energy fn. (Liapunov fn.) that always decreases each time the
network changes state.Eventually this in most reach a min m and stop ,thereby ensuing
that the net is stable .The Liapunov energy fn.
E = -0.5 ∑ ∑ wij yi yj - ∑ xjyj + ∑jyj
Where
E = an artificial network energy
Wij = wt from output of neuron I to the input of neuron j
өj
=
threshold of neuron j
Xj = external input to neuron j
Yj = output of neuron j
The change in energy E due to a change in the state of neuron j is given by:
∆ E = - [ ∑ (yi wij) + xj -өj ] ∆yj
= - [ y-inj - өj ] ∆yj
i≠j
15
The stability of such a net may be proven as follows :
1- Suppose that ( y.inj > Өj ) , then this will cause the term in brackets to be +Ve , Yj
must in the +Ve direction ( or remain constant ). Hence Δyj can only be +Ve or zero
and ΔE must be –Ve . That is the network energy must either decrease or stay
constant
2- Suppose that ( Y- inj < Өj) , then ΔYj can be only -Ve or zero , hence the energy
must decrease or stay constant.
3- Finally , if ( Y- injj= Өjj) , ΔYj is zero and the energy remains unchanged .
This shows that any change in the state of a new or will either reduce the energy or
maintain its cument value .Because the energy shows this continuous trend , eventually
it must find a minimum and stop . By definition such a network is stable.
Neusat Nets Based on Competition
Competition is the mechanism whereby the rmuctire of the net is such that the net is
forced to make a deasion as to which one unit will respond . in winnes take all only one
new or in the competing group will have a nor – zero output signal when the mpetition
is completed . An example is maxnet & the mexican hat . Some nets use fixed weights ,
however it is more usual to combine competition with some from of learning to adjust
the weights of the net . Some nets use unsuperwised learning where the net reeks to find
patterns or regularity in the input data .These types of nets are commonly used for
clustering . in a clustering net , the no. of input units is determined by the no . of
components in the input vector . However , each output unit represents a clusters that
can be formed . The weight vector for an output unit in a clustering net serves as a
representative or exemplar or &de – book for input patterns which the net has placed or
that cluster.During training , the net cument input vector . The weight vector for the
winner is then adjusted according to the learning algorithm .
16
Maxnet
This net is based or competition and can be used as a subnet to pick the node whose
input is the largest . The m nodes in this subnet are completely interconnected with
symmetric weights . The weights are fixed i.e. these is no training for the maxnet .
Procedure :
The activation fn . for the maxnet is
F (x) = x
if x > o
0
otherwise
Step o : Initialize activations & wts
Set(0 e <1 ∕m )
Aj(0) input to node Aj
1
if I = j
Wij =
+e
if i ≠ j
Step 1 : While stopping corditiori is false , do step 2 – 4
Step 2 :
Update the activation of each node
For j = 1 , 2….. m
Aj ( new ) = f [ aj ( old ) - ∑ a ( old ) ]
Step 3 :
Save activations for in use next iteration
Aj (old ) = aj (new )
Step 4 :
j = 1…... m
Test stopping condition
Note : in step 2 the input to the function is simply the total input to node A, from am
nodes including it serf . some precautions should be incorporated to handle the
17
situation in which two or more units have the same maximal , input if more than one
node has a non – zero activation , continue , otherwise stop .
Example :
Consider a maxnet with four neurons and inhibitory weights = +0.2 , given the
initial input signals :
aı (0) = 0.2 , a2 (0) = 0.4 , a3 (0) = 0.6 , a4 (0) = 0.8
The activations as the net iterates :A1 (1) = 0 , a2 (1) = 0.08 , a3 (1) = 0.32 , a4 (1) = 0.56
where say
a4 ( new ) = f [a4 (old ) - ∑ ak (old) ]
k≠j
= f [ 0.8 – 0.2 ( 0.2 + 0.4 + 0.6 ) ]
= f [ 0.8 – 0.2 * 1.2 ] = f [ 0.8 – 0.24 ]
= 0.56
A1 (2) = 0 , a2 (2) = 0 , a3 (2) = 0.192 , a4 (2) = 0.48
A1(3) = 0 , a2 (3) = 0 , a3 (3) = 0.096 , a4 (3) = 0.442
A1 (4) = 0 , a2 (4) = 0 , a3 (4) = 0.008 , a4 (4) = 0. 422
A1 (5) = 0 , a2 (5) = 0 , a3 (5) = 0 , a4 (5) = 0.421
N.B. : Some care should be taken to handle the situation in which two or more units
have the name maximal input
18
19