V I L L A N O V A U N I V E R S I T Y
Department of Electrical & Computer Engineering
ECE 8412 Introduction to Neural Networks
Homework 01 Due 13 September 2005
Chapters 1 and 2
Name: Santhosh Kumar Kokala
Complete the following and submit your work to me as a hard copy of a WORD document.
You can also fax your solutions to 610-519-4436.
01.
Which two neural network pioneers had the greatest impact in your opinion and explain why you
think so?
I think Warren McCulloch and Walter Pitts are the two pioneers who created a tremendous impact
in the field of neural networks. It’s only because of their work the modern view of neural networks
began in the 1940 and they were successful in showing that networks of artificial neurons could, in
principle, compute any arithmetic or logical function.
02.
List the names of the other pioneers in the development of neural networks and briefly state what each
Pioneer

Donald Hebb
Work
Proposed a mechanism for learning in
biological neurons

Frank Rosenblatt
Developed a perceptron network and
demonstrated its ability to perform
pattern recognition.

Bernard Widrow and Ted Hoff
Introduced a new learning algorithm
and used it to train adaptive linear
neural networks.

Teuvo Kohonen and James Anderson
Developed neural networks that could
act as memories.
did.
03.
Choose an application area listed in the chapter and give a URL that discusses a specific application in
that area. You must have an answer to this question that is different from all the other responses that I
receive.
Circuit Partitioning is one of the applications in the field of electronics. When given a set of circuit
elements and the net lists showing the connectivity between these elements, the objective of the
circuit partitioning is to allocate these elements in to two or more blocks such that the number of
external wires interconnecting these blocks is minimized.
URL: http://portal.acm.org/citation.cfm?id=74450
04.
Define the following terms:
- Neuron: A cell specialized to conduct and generate electrical impulses and to carry information
from one part of the brain to another.
- Dendrites: The dendrites are tree-like receptive networks of nerve fibers that carry electrical
signals in to the cell body.
- Cell body: The part of the cell that contains the nucleus and which effectively sums and
thresholds the incoming electrical signals from the dendrites.
- Axon: The axon is a single long fiber that carries the signal from the cell body out to other
neurons.
- Synapse: The point of contact between an axon of one cell and dendrite of another cell is called
a synapse.
- Response time of neurons: Time taken by the neuron to process the input and generate an output.
05.
If the brain were organized into 1,000,000 partitions how many neurons would be in each partition?
Show calculation!
Considering the fact that brain consists of approximately 1011 neurons then the number of neurons in
1,000,000 = 106 partitions is given by 1011/106 = 10(11-6) = 105 neurons per partition.
06.
Understanding the notation used in the text is critical. For starters complete the following table:
Mathematical Item
Possible Symbols
Scalars
Small italic letters: a, b, c
Vectors
Small bold nonitalic letters: a, b, c
Matrices
Capital BOLD nonitalic letters: A, B, C
07.
In the text, a vector is understood to be what kind of vector? row or column
In the text, a vector is considered as column vector.
08.
For a single-input neuron define or describe the following symbols:
Symbol Definition or Description
p
w
b
n
p is the scalar input of a neuron.
w is the adjustable scalar parameter which is multiplied by the input p and sent to the
summer.
b is the bias or offset which is another adjustable scalar parameter that is multiplied by the
other input 1 and sent to the summer.
n is nothing but the summer output, which is often referred to as the net input which goes
to the transfer function.
-2-
f
f is the transfer function or the activation function of the neuron that processes the
summer output.
a
a is the neuron output given by the activation function f.
09.
What is the output equation for a single-input neuron?
a = f (wp+b)
10.
What is the output equation for a multiple-input neuron?
a = f(Wp+b)
11.
Refer to Table 2.1.
a) Which transfer functions are linear or piece-wise linear?
Hard Limit, Symmetrical Hard Limit, Saturating Linear, Symmetric Saturating Linear, Positive
Linear are piece-wise linear functions.
b) Which transfer functions have outputs that range between 0 and +1?
Saturating Linear functions have outputs that range between 0 and +1.
c) Which transfer functions have outputs that range between -1 and +1?
Symmetric Saturating Linear functions have outputs that range
between -1 and +1.
d) Which transfer functions have a finite number of output values?
Hard Limit, Symmetrical Hard Limit and Competitive functions have finite number of output
values.
12.
In Figure 2.5 let
R =
f =
p1 =
p2 =
b =
w1,1
w1,2
2
tansig
1
0.5
- 0.2
= 0.1
= 0.05
Calculate the value of n and a.
Answer
a = f (Wp+b)
n = w1,1 p1 + w1,2 p2+…..+w1,R pR + b
n = 0.1 * 1 + 0.05 * 0.5 -0.2 = 0.1 + 0.025 - 0.2 = -0.075
so a = f (Wp+b) = f (n) = tansig (n) = tansig (-0.075) = -0.0749
-3-
13.
Explain the differences between Figure 2.7 and Figure 2.8 by relating the symbols in Figure 2.8 to the
symbols in Figure 2.7.
Figure 2.8 can be considered as an abbreviated notation of the figure 2.7. In the figure 2.7 the
inputs are represented by p1, p2........pR these inputs are shown as vector p of dimension R x 1.
The weights w1,1 , w1,2 .........w1,R are shown as matrix W of dimension S x R
Similarly the biases and outputs for each neuron are shown as vector b and a of dimension S x 1.
14.
If the weight matrix in equation (2.6) has dimension 4 x 5,
a) How many neurons are there and why?
There are 4 neurons and this can deduced from the dimension of the weight matrix. Generally
the weight matrix is given by WS X R = W4 X 5 where S is the number of neurons.
b) How many inputs are there and why?
Number of inputs to are 5 and this is given by WS X R = W4 X 5 where R is the number of inputs.
c) Element wi,j connects which input to which neuron and how do you know?
The row indices of the elements of the matrix W indicate the destination neuron associated with
that weight, while the column indicates the source of the input for that weight. Thus the indices
in wi, j say that this weight represents the connection to the ith neuron from the jth source.
15.
In Figure 2.8 let
p=
0.4447
0.6154
0.7919
b=
4.6091
3.6910
0.8813
W = 0.4057
0.9355
0.9169
0.4103
0.8936
0.0579
0.3529
0.8132
0.0099
f = poslin
Calculate the output a.
Answer
a = f (Wp+b)
Wp+b =
0.4057 0.4103 0.3529
0.4447  4.6091
0.9355 0.8936 0.8132 * 0.6154  + 3.6910 



 

0.9169 0.0579 0.0099
0.7919  0.8813
-4-
=
0.7124 4.6091
0.7124  4.6091 5.3215
1.6099  + 3.6910  = 1.6099  3.6910  = 5.3009

 


 

0.4512 0.8813
0.4512  0.8813 1.3325 
5.3215
So a = f (Wp+b) = poslin ( 5.3009 ) =


1.3325 
16.
5.3215
5.3009


1.3325 
In Figure 2.10 let
p=
0.9501
0.2311
0.6068
b1 = b2 = b3 = 4.6091
3.6910
0.8813
W1 = W2 = W3 = 0.4057
0.9355
0.9169
f11  logsig
0.4103
0.8936
0.0579
0.3529
0.8132
0.0099
f12  tansig
f13  satlins
Calculate a1, a2, and a3.
Answer
0.4057 0.4103 0.3529
a1 = f11 (W1p + b1) = logsig ( 0.9355 0.8936 0.8132 *


0.9169 0.0579 0.0099
5.3035
= logsig ( 5.2798 ) =
1.7718 
0.9501
0.2311 +


0.6068
4.6091
3.6910  )


0.8813
0.9951 
0.9949 


0.8547 
0.4057 0.4103 0.3529 0.9951  4.6091
a2 = f12(W2 a1 + b2) = tansig( 0.9355 0.8936 0.8132 * 0.9949  + 3.6910  )

 
 

0.9169 0.0579 0.0099 0.8547  0.8813
9.5523 
= tansig ( 4.8113 ) =


6.0673
1.0000 
0.9999 


1.0000 
-5-
a3
= f1
3
(W3
a2
+
b3)
0.4057 0.4103 0.3529 1.0000 4.6091
= satlins( 0.9355 0.8936 0.8132 * 1.0000 + 3.6910  )




0.9169 0.0579 0.0099 1.0000 0.8813
16.7027 
= satlins ( 30.7965 ) =
10.5575 
17.
1
1

1
In Figure 2.11 let
u(t) = 1
1
for t ≥ 0, and
a(0) = - 2
-2
Calculate a(t) for t = 0, 1, 2, and 3. Show how you did it.
Answer:
  2
For t=0 it’s given already a(0) =  
  2
1
For t =1 a(1) = u(1-1) = u(0) =  
1
1
For t =2 a(2) = u(2-1) = u(1) =  
1
1
For t =3 a(3) = u(3-1) = u(2) =  
1
-6-
18.
The network in Figure 2.13 is a discrete-time recurrent network.
a) What makes it discrete-time?
In the Figure 2.13 the recurrent network uses Symmetric Saturating Linear function as
activation function of the system
a (t+1) = satlins (W a(t) + b)
This transfer function calculates a layer’s output from its net input and returns values of N
truncated in to the interval [-1, 1]. Hence the network in the Figure 2.13 can be called as a
discrete time recurrent network.
b) What makes it recurrent?
The network in the figure 2.13 has backward connections i.e. some of its outputs are connected
to its inputs. This property of the network makes it recurrent.
19.
A two layer neural network is to have four inputs and six outputs.
The range of the outputs is to be continuous between 0 and 1.
a) How many neurons would you use in the first layer?
The above problem specifications does not mention the number of neurons in the hidden layer
i.e. in the first layer.
b) How many neurons would you use in the second layer?
Since we need six outputs six neurons are required in the second layer.
c) What is the dimension of the first layer weight matrix?
Dimension of first layer weight matrix cannot be calculated as it depends on the number of first
layer neurons.
d) What is the dimension of the second layer weight matrix?
Dimension of the second layer weight matrix cannot be determined since the problem doesn’t
specify the number of outputs from the first layer which are nothing but the inputs to the second
layer.
e) What kind of transfer function would you use in the first layer?
Satlins, logsig, tansig, purelin
e) What kind of transfer function would you use in the second layer?
Since the output has to be continuous between 0 and 1 we use Saturating Linear function.
e) Why would it be better to use biases in each layer?
The bias gives the network an extra variable through which we can control the network varying
it. To avoid zero output when there are no inputs we can use bias.
-7-