Introduction to Neural Networks
Homework 8
Student: Dazhi Li
Student ID: 153045
E13.5
The instar shown in Figure E13.2 is to be used to recognize a vector.
i.
Train the network with the instar rule on the following training sequence.
Apply the instar tule to the second input's weights only (which shoud be
initialized to zeros), using a learning rate of 0.6.The other weight and the bias
are to remain constant at the values in the figure.
ii.
What were your final values for W?
iii.
How do these final values compare with the vectors in the training sequence?
iv.
What magnitude woud you expect the weights to have after traning, if the
network were trained for many more iterations of the same training sequence?
i.
w0 = 1;
W = [0 0];
b = -0.5;
lr = 0.6;
for i = 1 : 6
fprintf('Interation %d\n', i);
p0 = MOD(i, 2);
p = [hardlims(p0-0.5)*0.174; 0.985];
a = hardlim(w0 * p0 + W * p + b)
W = W + lr * a * (p' - W)
end
Interation
a =
1
W =
0.1044
Interation
a =
1
W =
-0.0626
Interation
a =
1
W =
0.0793
Interation
a =
1
W =
-0.0727
Interation
a =
1
W =
1
0.5910
2
0.8274
3
0.9220
4
0.9598
5
0.0753
Interation 6
a =
1
W =
-0.0743
ii. Finally,
0.9749
0.9810
W = [-0.0743 0.9810]
iii. The weight's final value is either [0.0746 0.9850] or [-0.0746 0.9850], which is
determined by the last one of the inputs.
iv. The magnitude of the w when the network is trained for many more iterations of
the same training sequence will not exceed 2 . This is because the values of the
inputs have been nomalized into [-1 1], so the weights when the network is trained for
many more iterations will also be nomalized. And the truth is the magnitude of the
weight is 0.9878 < 2
E14.2
Consider the input vectors and initial weights shown in Figure E14.1
i.
Draw the diagram of a competitive network that could cassify the data above
so that each of the three clusters of vectors would have its own class.
ii.
Train the network graphicaly (using the initial weights shown) by presenting
the labeled vectors in the following order:
p1, p2, p3, p4
Recall that the competitive trasfer function chooses the neuron with the lowest
index to win if more than one neuron has the same net input. The Kohonen
rule is introduced graphicaly in Figure 14.3
iii.
Redraw the diagram in Figure E14.1, showing your final weight vectors and
the decision boundaries between each region that represents a class.
i.
W = [ 0 1; 1 0; -1 0]
Inputs
p
Hard Limit Neuron
W
2X1
3X2
n
3X1
a = compet(Wp)
C
a
3X1
ii.
w1 = [-0.5; -0.866];
w2 = [0.866; -0.5];
w3 = [0.9682; -0.25];
W = cat(1, w1', w2', w3');
p1 = [0; 1];
p2 = [-1; 0];
p3 = [1; 0];
p4 = [-0.25; 0.9682];
input = cat(2, p1, p2, p3, p4);
lr =0.5;
for i = 1:4
n= mod(i,4);
if n==0
n=4;
end
p = input(:, n);
a = compet(W * p)
if a(1,1) == 1
w1 = w1 + lr * (p W(1, :) = w1';
elseif a(2,1) == 1
w2 = w2 + lr * (p W(2, :) = w2';
elseif a(3,1) == 1
w3 = w3 + lr * (p W(3, :) = w3';
end
W
subplot(2, 2, i);
axis([-1 1 -1 1]);
hold on;
x = [0 w1(1)];
y = [0 w1(2)];
plot(x, y, 'r');
x = [0 w2(1)];
y = [0 w2(2)];
plot(x, y, 'b');
x = [0 w3(1)];
y = [0 w3(2)];
plot(x, y, 'g');
plot(p(1), p(2), 'o');
title(['Present vector
end
a =
(3,1)
1
W =
-0.5000
0.8660
0.4841
-0.8660
-0.5000
0.3750
a =
(1,1)
1
w1);
w2);
w3);
p',num2str(i)]);
W =
-0.7500
0.8660
0.4841
-0.4330
-0.5000
0.3750
a =
(2,1)
1
W =
-0.7500
0.9330
0.4841
-0.4330
-0.2500
0.3750
a =
(3,1)
1
W =
-0.7500
0.9330
0.1170
-0.4330
-0.2500
0.6716
Present vector p1
Present vector p2
1
1
0.5
0.5
0
0
-0.5
-0.5
-1
-1
-0.5
0
0.5
1
-1
-1
Present vector p3
1
0.5
0.5
0
0
-0.5
-0.5
-0.5
0
0
0.5
1
Present vector p4
1
-1
-1
-0.5
0.5
1
-1
-1
-0.5
0
0.5
1
In the figure above, 1w is presented in red, 2w is presented in blue, 3w is presented in
green.
p4
iii.
p1
3w
p3
p2
1w
2w
E14.4
Figure E14.2 show a competitive network with biases. A typical learning rule for the bias
bi of neuron i is
binew = 0.9 biold, if i <> i*
binew = biold – 0.2, if i = i*
i.
Examine the vectors in Figure E14.3. Is there any order in which the vectors
can be presented that will cause 1w to win the competition and move closer to
one of the vectors? (Note: assume that adaptive biases are not being used.)
ii.
Give the input vectors and the initial weights and biases defined below,
calculate the weights (using the Kohonen rule) and the biases (using the above
bias rule). Repeat the sequence shown below until neuron 1 wins the
competition.
p1=[-1; 0], p2=[0; 1], p3=[1/sqrt(2) 1/sqrt(2)]
1w=[0; -1], 2w=[-2/sqrt(5) -1/sqrt(5)], 3w=[-1/sqrt(5) -2/sqrt(5)]
Sequence of input vectors: p1, p2,p3, p1, p2, p3…
iii.
How many presentations occur before 1w wins the competition?
i. If adaptive biases are not used, neuron 1 will never win. Since neuron 1 is always
farther than neuron 2 and 3 from any input vector. 2w finally converges to p1. 3w
oscillates between p2 and p3.
ii.
w1 = [0; -1];
w2 = [-2/sqrt(5); -1/sqrt(5)];
w3 = [-1/sqrt(5); -2/sqrt(5)];
W = cat(1, w1', w2', w3');
p1 = [-1; 0];
p2 = [0; 1];
p3 = [1/sqrt(2); 1/sqrt(2)];
input = cat(2, p1, p2, p3);
bias = [1; 1; 1];
lr =0.5;
winner = 0;
n = 0;
while winner ~= 1
i = mod(n,3)+1;
n = n + 1;
p = input(:, i);
a = compet(W * p + bias);
if a(1,1) == 1
w1 = w1 + lr * (p - w1);
W(1, :) = w1';
winner = 1;
elseif a(2,1) == 1
w2 = w2 + lr * (p - w2);
W(2, :) = w2';
winner = 2;
elseif a(3,1) == 1
w3 = w3 + lr * (p - w3);
W(3, :) = w3';
winner = 3;
end
for b = 1:3
if b == winner
bias(b) = bias(b) - 0.2;
else
bias(b) = bias(b) * 0.9;
end
end
if winner == 1
break;
end
end
W
bias
fprintf('Neuron 1 winned after %d presentations.\n', n);
W =
0.3536
-0.1464
0.2355
0.9022
-0.9914
-0.0140
bias =
-0.0784
-1.3155
-0.4059
Neuron 1 winned after 21 presentations.
iii. 21 presentations occur before 1w wins the competition.
E14.8
We would like a classifier that divides the interval of the input space defined below into
five classes.
0 <= p1 <= 1
i.
Use MATLAB to randomly generate 100 values in the interval shown above
with uniform distribution.
ii.
Square each number so that the distribution is no longer uniform.
iii.
Write a MATLAB M-file to implement a competitive layer. Use the M-file to
train a five-neuron competitive layer on the squared values until its weights
are fairly stable.
iv.
How are the weight values of the competitive layer distributed? Is there some
relationship between how the weights are distributed and how the squared
input values are distributed?
p = rand(1, 100);
p2 = (p .* p);
plot(p2(1, : ), '+r');
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
10
20
30
40
net = newc([0 1], 5);
net.trainParam.epochs = 1000;
net = train(net, p2);
figure;
hold on;
h = plot(net.iw{1}(:, 1), 'ob');
net.iw{1}
figure(2);
subplot(1, 2, 1), hist(p2);
subplot(1, 2, 2), hist(net.iw{1});
ans =
0.7690
0.1559
0.1882
0.4978
0.2470
50
60
70
80
90
100
25
2
1.8
20
1.6
1.4
15
1.2
1
10
0.8
0.6
5
0.4
0.2
0
0
0.5
1
0
0
0.2
0.4
0.6
0.8
W = [0.7690 0.1559 0.1882 0.4978 0.2470]
The distribution of the weights is also non-uniform. From the figure above, it can be seen
that the distribution of weights represents the convex parts of the input value.