NEURAL NETWORKS
Based on sets of notes prepared by Dr. Patrick H. Corr, Brendan Coburn and John
Gilligan
Neural Networks (also known as Connectionist models or Parallel Distributed
Processing models) are information processing systems which model the brain’s
cognitive process by imitating some of its basic structures and operations. Interest in
these networks was originally biologically motivated. They were developed with the
expectation of gaining new insight into the workings of the brain. In the 1940’s and
1950’s there was a certain amount of success in the research and development of
neural networks but eventually the attraction of these systems declined due to a
number of factors. However, within the past decade interest in neural networks has
been revived. Although these networks are still helpful in the research of the brain’s
cognitive process, today they are actually implemented in the processing of
information.
In 1943 two scientists, Warren McCulloch and Walter Pitts, proposed the first
artificial model of a biological neuron [McC]. This synthetic neuron is still the basis
for most of today’s neural networks.
Rosenblatt came up with his two layered perceptron which was subsequently shown
to be defective by Papert and Minsky which lead to a huge decline in funding and
interest in Neural Networks.
2.3.5 Other Developments
During this period, even though there was a lack of funding and interest in neural
networks, a small number of researchers continued to investigate the potential of
neural models. A number of papers were published, but none had any great impact.
Many of these reports concentrated on the potential of neural networks for aiding in
the explanation of biological behaviour (e.g. [Mal], [Bro], [Mar], [Bie], [Coo]).
Others focused on real world implementations. In 1972 Teuvo Kohonen and James A.
Anderson independently proposed the same model for associative memory [Koh],
[An1] and in 1976 Marr and Poggio applied a neural network to a realistic problem in
computational vision, stereopsis [Mar]. Other projects included [Lit], [Gr1], [Gr2],
[Ama], [An2], [McC].
2.4.1 The Discovery of Backpropagation
The backpropagation learning algorithm was developed independently by Rumelhart
[Ru1], [Ru2], Le Cun [Cun] and Parker [Par] in 1986. It was subsequently discovered
that the algorithm had also been described by Paul Werbos in his Harvard Ph.D thesis
in 1974 [Wer]. Error backpropagation networks are the most widely used neural
network model as they can be applied to almost any problem that requires pattern
mapping. It was the discovery of this paradigm that brought neural networks out of
the research area and into real world implementation.
What and why?
Neural Networks: a bottom-up attempt to model the
functionality of the brain.
Two main areas of activity:


Biological
o Try to model biological neural systems
Computational
o Artificial neural networks are
biologically inspired butnot necessarily
biologically plausible
o So may use other terms:
Connectionism, Parallel Distributed
Processing, Adaptive Systems Theory.
A simplified view of a neuron is shown in the diagram below.
Signals move from neuron to neuron via electrochemical reactions. The synapses
release a chemical transmitter which enters the dendrite. This raises or lowers the
electrical potential of the cell body.
The soma sums the inputs it receives and once a threshold level is reached an
electrical impulse is sent down the axon (often known as firing).
These impulses eventually reach synapses and the cycle continues.
Synapses which raise the potential within a cell body are called excitatory. Synapses
which lower the potential are called inhibitory.
It has been found that synapses exhibit plasticity. This means that long-term changes
in the strengths of the connections can be formed depending on the firing patterns
of other neurons. This is thought to be the basis for learning in our brains.
Marking Scheme
1 X 5 basic components
2 for diagram.
(c ) How is the neuron modelled in Artificial Neural Nets?
(6 marks)
Modelling a Neuron
To model the brain we need to model a neuron. Each neuron performs a simple
computation. It receives signals from its input links and it uses these values to
compute the activation level (or output) for the neuron. This value is passed to other
neurons via its output links.
The input value received of a neuron is calculated by summing the weighted input
values from its input links. That is
in i   j Wj , iaj
An activation function takes the neuron input value and produces a value which
becomes the output value of the neuron. This value is passed to other neurons in the
network.
This is summarised in this diagram and the notes below.
aj
:
Activation value of unit j
wj,i
:
Weight on the link from unit j to unit i
ini
:
Weighted sum of inputs to unit i
ai
:
Activation value of unit i (also known as the output value)
g
:
Activation function
Or, in English.
A neuron is connected to other neurons via its input and output links. Each incoming
neuron has an activation value and each connection has a weight associated with it.
The neuron sums the incoming weighted values and this value is input to an activation
function. The output of the activation function is the output from the neuron.
Some common activation functions are shown below.
These functions can be defined as follows.
Stept(x)
=
1 if x >= t, else 0
Sign(x)
=
+1 if x >= 0, else –1
Sigmoid(x) =
1/(1+e-x)
On occasions an identify function is also used (i.e. where the input to the neuron
becomes the output). This function is normally used in the input layer where the
inputs to the neural network are passed into the network unchanged.
Interests in neural network differ according to profession.
Neurobiologists and psychologists
understanding our brain
Engineers and physicists
a tool to recognise patterns in noisy data (see Ts at right)
Business analysts and engineers
a tool for modelling data
Computer scientists and mathematicians
networks offer an alternative model of computing: machines that may be
taught rather than programmed
Artificial Intelligensia, cognitive scientists and philosophers
Subsymbolic processing (reasoning with patterns, not symbols)
Symbol
Subsymbol
top-down
bottom-up
Explicit
implicit
Rules
examples
Serial
parallel
digital/boolean (true or false)
analog/fuzzy
Brittle
robust
Some Application Areas
A good overview of NN applications is provided on the pages set up by the DTI
Neural Applications programme and the later Smart software for decision makers
programme


NCTT programme
Smart software for decision makers
I've combined their overview with Biggus's classification of NN applications in
further notes on neural network applications.
Neural computing provides an approach which is closer to human perception and
recognition than traditional computing.
Neural computing systems are adept at many pattern recognition tasks, more so than
both traditional statistical and expert systems.
Combinatorial Problems
Neural computing systems have shown some promise for solving 'NP-complete'
problems.
In solving this type of problem neural networks offer the facility to find a 'dirty'
solution quickly rather than using significantly more resources to find the optimal
solution for little extra gain.
The multi-layer perceptron has been applied to a wide variety of problems calling for
a non-linear mapping between input and output.
NETTALK AND DECTALK
DECTALK is a system developed by DEC which reads English characters and
produces, with a 95% accuracy, the correct pronunciation for an input word or text
pattern.


>DECTALK is an expert system which took 20 years to finalise.
It uses a list of pronunciation rules and a large dictionary of exceptions.
NETTALK, a neural network version of DECTALK, was constructed over one
summer vacation.


After 16 hours of training the system could read a 100-word sample text with
98% accuracy!
When trained with 15,000 words it achieved 86% accuracy on a test set.
NETTALK is an example of one of the advantages of the neural network approach
over the symbolic approach to Artificial Intelligence.


It is difficult to simulate the learning process in a symbolic system; rules and
exceptions must be known.
On the other hand, neural systems exhibit learning very clearly; the network
learns by example.
COMPARISONS
Neural networks are not a panacea for problems in information processing which
traditional methods find difficult or complex.
The neural approach is simply an alternative paradigm.
ATTRACTIVE PROPERTIES OF NEURAL NETWORKS
Parallelism
Neural Networks are inherently parallel and naturally amenable to expression in a
parallel notation and implementation on parallel hardware.
Capacity for Adaptation
In general, neural systems are capable of learning.
Some networks have the capacity to self-organise, ensuring their stability as dynamic
systems.
A self-organising network can take account of a change in the problem that it is
solving, or may learn to resolve the problem in a new manner.
Distributed Memory
In neural networks 'memory' corresponds to an activation map of the neurons.
Memory is thus distributed over many units giving resistance to noise.
In distributed memories, such as neural networks, it is possible to start with noisy data
and to recall the correct data.
Fault Tolerance
Distributed memory is also responsible for fault tolerance.
In most neural networks, if some PEs are destroyed, or their connections altered
slightly, then the behaviour of the network as a whole is only slightly degraded.
The characteristic of graceful degradation makes neural computing systems
extremely well suited for applications where failure of control equipment means
disaster.
Capacity for Generalisation
Designers of Expert Systems have difficulty in formulation rules which encapsulate
an experts knowledge in relation to some problem.
A neural system may learn the rules simply from a set of examples.
The generalisation capacity of a neural network is its capacity to give a satisfactory
response for an input which is not part of the set of examples on which it was trained.
The capacity for generalisation is an essential feature of a classification system
Certain aspects of generalisation behaviour are interesting because they are intuitively
quite close to human generalisation.
Ease of Construction
Computer simulations of small applications can be implemented relatively quickly.
LIMITATIONS IN THE USE OF NEURAL NETWORKS





Neural systems are inherently parallel but are normally simulated on a
sequential machines.
o Processing time can rise quickly as the size of the problem grows - The
Scaling Problem
o However, a direct hardware approach would lose the flexibility offered
by a software implementation.
o In consequence, neural networks have been used to address only small
problems.
The performance of a network can be sensitive to the quality and type of
preprocessing of the input data.
Neural networks cannot explain the results they obtain; their rules of operation
are completely unknown.
Performance is measured by statistical methods giving rise to distrust on the
part of potential users.
Many of the design decisions required in developing an application are not
well understood.
Comparison of neural techniques and symbolic artificial
intelligence
Early work on Neural systems was largely abandoned after serious limitations of
earlier models were highlighted in 1969.
Growth of Artificial Intelligence based on the hypothesis that thought processes could
be modelled using a set of symbols and applying a set of logical transformation rules.
The symbolic approach has a number of limitations:








It is essentially sequential and difficult to parallelise
When the quantity of data increases, the methods may suffer a combinatorial
explosion.
An item of knowledge is represented by a precise object, perhaps a byte in
memory, or a production rule. This localised representation of knowledge does
not lend itself to a robust system.
The learning process seems difficult to simulate in a symbolic system.
The connectionist approach offers the following advantages over the symbolic
approach:
parallel and real-time operation of many different components
the distributed representation of knowledge
learning by modifying connection weights.
Both approaches are likely to be combined in the future. For now, here are some rules
of thumb for choosing the approach to use:
Input -> Output
Facts -> Decision
Model:
Logic representation
(facts and rules)
of human
expertise
Expert system
learned from
data
Machine
induction
Numbers (measurements ->
Decision
Neural
Mathematical calculation
predictions)
support system network
Multiple Layer Feed Forward Networks
Solving non linearly separable problems
As pointed out before, XOR is an example of a non linearly separable problem which
two layer neural nets cannot solve. By adding another layer, the hidden layer, such
problems can be solved.
Solving XOR with NETS
With 3 hidden nodes in one hidden layer
- xor3.net
Example weight file - xor3.pwt
With 1 hidden node xor31.net
Example weight file - xor31.pwt
In xor31.net you can roughly trace how the trained net works. Assume nodes have
step function thresholds at 0.0 and output 0 or 1. (The actually use the sigmoid
function.)
If you apply x1= 0 and x2 = 0, no signal arrives at the middle node. Since the bias is
2.2 the node outputs 1 through the -6.51 weight. The outer connections also receive
no inputs, so the output node receives nothing from those lines. The -6.51 signal
outweighs the 3.93 bias so there is no output at y, a correct result.
If x1 =1 and x2 = 1, then the weighted inputs into the centre node overwhelm the 2.12
bias on the node and it produces no output along the line weighted -6.51. Similarly,
along the outer lines the weighted input to the output node is 1 * - 2.64 + 1 * -2.66 = 5.3. This overwhelms the 3.93 bias so that the net input to the node is -1.37 which is
below the 0 threshold, so there is no output at y. This is also correct.
If x1 = 1 and x2 = 0, the net input to the centre node, including the bias is only -3.24,
well below 0.0 so it does not fire. Therefore there is no input to the output node along
the center connection.
On the left outer line we have an input of 1 * -2.64. On the right outer line the input is
0 to the output node. The bias is 3.93 so the net input to the output node is 3.03- 2.64
> 0.0 do the node fires and an output of 1 appears at y.
If x1 = 0 and x2 = 1, the result is similar to x1 = 1 and x2 = 0. In all cases the net
correctly implements the XOR function.
It turns out that 3 layer or 4 layer nets can mimic any computable function. Thus they
have the computing power of a Turning Machine. In other words, they are as powerful
as any computational device can be.
One might ask, just what is the net learning, and how is it trained?
Features
People have tried, with limited success, to better understand these networks with
hidden layers by associating the hidden nodes with features of the input data. This
diagram illustrates the idea of feature detection.
The idea is that each of the two hidden nodes becomes associated with a feature of the
input data set. The following sketches illustrate what might happen.
Don't take the lines too seriously - they are just for visual effect. The first input
pattern is 0101010 which produces output 100. Similarly 0100010 ==> 010, and
0001000 ==> 001.
Diagram (b) shows the input and output nodes. But what about the 'hidden' nodes?
They might look like this
Fig. (c) represents the features detected by the network's hidden nodes. The points
represent the weights along the connections coming from the 7 input nodes to each of
the hidden nodes.
The connections from the hidden nodes to the output nodes combine these features to
produce the appropriate output. The joining of the points is meant to provide an aid in
visualizing how the features 'add up' to produce the output.
Many people take this feature analysis 'with a grain of salt', and prefer to visualize the
network as a black box which, when trained, does what it supposed to.
Training multilayer networks with back propagation.
The discovery of the backpropagation algorithm lead to an explosion of interest in
Neural Networks. Feed forward multilayer networks trained with the backpropagation
algorithm are still the most common kind today. How does the algorithm work?
The big picture
Calculating errors for a 3 layer network
First of all, the appropriate error signals must be calculated.
or, more simply,
Here f(I) is the threshold function. The total weighted signal (sometimes called
'activation') into a node is represented by I. The y's are outputs from various nodes.
The index, j, numbers output nodes. Index, i, numbers the hidden nodes (middle
layer). w(i, j) is the weight on the line going from node i in the hidden layer to node j
in the output layer. It is assumed in the second set of equations that f(I) is the sigmoid
function.
The errors at the output nodes is smoothed by multiplying them the derivative of the
sigmoid function,
.
This has the effect of reducing the effects of larger errors.
Once the errors at the output nodes are calculated, they are treated as inputs starting
from the output node. The net is run 'backwards' as the error signals are propagated
from the output node layer towards the input node layer. This backward error
propagation (backward in the sense that the error signals flow in the opposite
direction the direction of the activations caused by the normal inputs) enables the
calculation of errors at the hidden layer corresponding to the output errors.
The second or fourth of the above equations determines the errors at the hidden nodes.
The calculation is similar to the delta rule.
The error at each hidden node, j, coming from the output node, i, is proportional to the
weight of the line from i to j and to the error at output node, j (calculated using the
first or third equation). The total error at hidden node i is the sum of the errors coming
from all the output nodes receiving activation from node i. Once again, the derivative
of the sigmoid function smooths the results by diminishing the effects of large errors.
Using the errors to adjust the weights
Once the errors are known, the weights can be adjusted, layer by layer. The weights of
the hidden nodes are also adjusted correctly. The formula is,
where beta is the learning rate.
The correction works layer by layer, backwards, from output layer to input layer. In
the 3 layer (of nodes) net there are only two layers of weights.
The errors at the output nodes are used to calculate the new weights between the
hidden (middle) layer and the output layer. Then the delta formula is used again to
calculate the new weights between the input and the hidden layer using the errors at
the middle layer derived using the 2nd o r 4th error equations above.
Here is an attempt to put it all together. The notation is slightly different from the
above notation.
2.3.4 Linear Separability and Minsky and Papert’s Perceptrons
During the years following the introduction of the perceptron and the ADALINE,
research in neural networks prospered. However, both Rosenblatt’s network and
Widrow and Hoff’s adaptive neurons suffered from one major problem. To illustrate
this problem, consider the way the neurons in both models compute their net inputs.
The equation used is
n
I =  wi xi
i 1
This can rewritten (for a two dimensional input pattern) as
I  x1w1  x2w2
This equation is also used to calculate the dot product of two vectors in a Cartesian
co-ordinate system. Therefore what is really happening is the dot product between the
weight vector and the input vector is being calculated. Another form of this equation
is
x  w | x|| w| cos 
Since the length of a vector must always be positive, the only term in the above
equation that can affect the sign of the result is the cosine of the angle between the
two vectors. The dot product will be positive if the angle between the vectors is less
than 90 and negative if the angle is greater than 90. It can be seen from figure 2.6
that the ADALINE (the outputs used in the figure are for the ADALINE. The only
difference with the perceptron is that the outputs would be +1 or 0) produces an
output of +1 for input patterns that are within 90 of its weight vector and a -1 for all
other patterns. Therefore in order for the ADALINE or perceptron to successfully
solve a particular problem, that problem must be linearly separable, i.e. the
categories of the inputs can be separated by a straight line. This is a simple
representation of this problem, but even so, it is easy to see that the capabilities of
these neural models were extremely limited since the majority of problems do not
meet this criterion (the XOR problem being a classic example [Day]).
Class A (+1 Output)
Minsky and Papert, Perceptrons
In the years leading up to the publication
Weight vector
of this book, the appeal of neural
networks had begun to dwindle. There Class B (-1 Output)
were a number of reasons for this decline
in interest. Firstly, neural networks failed
Figure 2.6
to meet the expectations of the scientific
Separation of two different classes
community. Although originally greeted
with great excitement, no major progress was made in the follow up to their
discovery, and excessive hype put pressure the developers of these networks to
produce results quickly. Another obstacle that neural networks had to overcome was
the apprehension which inevitably meets any attempt to develop artificially intelligent
machines. There was constant resistance to the idea of building a ‘chunk of brain’.
Rosenblatt once quoted a headline from an Oklahoma newspaper [Ros2] which said
“Frankenstien Monster Designed by Navy Robot That Thinks.” The new field of
Artificial Intelligence also lessened the attraction of neural computing as it seemed
potentially more successful in the design of machine intelligence.
All these factors contributed to the decline of neural networks. However, the final
blow came in 1969 with the arrival of the book Perceptrons by Marvin Minsky and
Seymour Papert [Min]. This book highlighted many difficulties with the existing
neural models, particularly the problem of linear separability. Although this problem
was already recognised, its full extent was never really realised. By using
mathematical analysis, Minsky and Papert showed that one and two layered neural
networks would only ever be able to solve linearly separable problems and therefore
their usefulness was extremely limited. This book all but wiped out funding and
interest in neural computation. To most scientists it seemed that these networks were
a lost cause and the majority of the scientific community turned their attention
elsewhere, in particular to the field of Artificial Intelligence.
2.4.2 An Overview of Backpropagation Networks
In their book Perceptrons, Minsky and Papert concluded that any network with only
one layer of adjustable weights would only ever be able to solve linear separable
problems. They also concluded that no algorithm could ever be constructed that
would allow the proper modification of a two or more layers and therefore the
potential of neural networks was extremely limited. However, with the discovery of
backpropagation, this obstacle was overcome.
Backpropagation Network Architecture
A backpropagation network typically consists of three or more layers of nodes. The
first layer is the known as the input layer and the last layer is known as the output
layer. Any layers of nodes in between the input and output layers are known as
hidden layers. Each unit in a layer is connected to every unit in the next layer. There
are no interlayer connections. The operation of the network consists of a forward pass
I
E
N
R
P
R
U
O
T
R
Output Layer
Hidden Layer
Input Layer
Figure 2.7
A Backpropagation Network
of the input through the network and then a backward pass of an error value which is
used in the weight modification (figure 2.7).
Forward Propagation
A forward propagation step is initiated when an input pattern is presented to the
network. No processing is performed at the input layer. The pattern is propagated
forward to the next layer, and each node in this layer performs a weighted sum on all
its inputs (as in the perceptron and ADALINE). After this sum has been calculated, a
function is used to compute the unit’s output. The function used to perform this
operation is the sigmoid function,
f ( x) 
1
1  e x
The main reason why this particular function is chosen is that its derivative, which is
used in the learning law, is easily computed. The result obtained after applying this
function to the net input is taken to be the node’s output value. This process is
continued until the pattern has been propagated through the entire network and
reaches the output layer. The activation pattern at the output layer is taken as the
network’s result.
Backward Propagation
The first step in the backpropagation stage is the calculation of the error between the
network’s result and the desired response. This occurs when the forward propagation
phase is completed. Each processing unit in the output layer is compared to its
corresponding entry in the desired pattern and an error is calculated for each node in
the output layer. The weights are then modified for all of the connections going into
the output layer. Next, the error is backpropagated to the hidden layers and by using
the generalised delta rule, the weights are adjusted for all connections going into the
hidden layer. The procedure is continued until the last layer of weights have been
modified. The forward and backward propagation phases are repeated until the
network’s output is equal to the desired result.
The Backpropagation Learning Law
The learning law used in the backpropagation network is a modification of the
original delta rule used in the ADALINE. The new form, known as the Generalised
Delta Rule, allows for the adjustment of the weights in the hidden layer, a feat
deemed impossible by Minsky and Papert. It uses the derivative of the activation
function of nodes (which in most cases is the sigmoid function) to determine the
extent of the adjustment to the weights connecting to the hidden layers. A full
mathematical description of the generalised delta rule can be found in [Bra].
2.4.2 Components of a Backpropagation Network
The Input Layer
The input layer of a backpropagation network acts solely as a buffer to hold the
patterns being presented to the network. Each node in the input layer corresponds to
one entry in the pattern. No processing is done at the input layer. The pattern is fed
forward from the input layer to the next layer.
The Hidden Layers
It is the hidden layers which give the
backpropagation network its exceptional
computation abilities. The units in the
hidden layers act as “feature detectors”.
They extract information from the input
patterns which can be used to
distinguish between particular classes.
The network creates its own internal
representation of the data. An example
of this feature detection is given in
[Day]. A backpropagation network with
Figure 2.8
Example Network Training Set
one hidden layer comprising of two units was trained on the patterns shown in figure
2.8. The patterns were chosen so that the distinguishing features of each pattern were
obvious. From figure 2.8, it can be seen
that only two feature detectors are
needed, the second and third patterns are
combined in the first. Thus, if feature
detectors are organised to respond to the
second and third patterns, then the first
pattern can be identified when both
feature detectors are activated. After
training, the features encoded by the
network were found by reading weight
Figure 2.9
Feature
Detectors
values from the trained network. The
weights were read from the first layer of weights, for connections that originated at
the input layer and terminated at the two hidden “feature detector” units. Graphs of
these weights were used to visually represent the features to which each of the hidden
units responded. It is apparent from figure 2.9 that these features matched the
contours of the second and third training patterns, providing distinguishing
characteristics for all three patterns.
The Output Layer
The output layer of a network uses the response of the feature detectors in the hidden
layer. Each unit in the output layer emphasises each feature according to the values of
the connecting weights. The pattern of activation at this layer is taken as the
network’s response.
Summary
The backpropagation neural network is by the far the most powerful and adaptive
neural model available. It is an excellent choice for any form of pattern mapping
problems. For this study, backpropagation networks were chosen because of their
effectiveness and versatility.
(10 marks)
d) Calculate the weight adjustments in the following network for expected outputs
of {1,1} and the learning rate is 1:
(9 marks)
The Target Values are 1, 1 and the learning rate is 1
Use F(X) as the Activation Function 1 / (1 + e -x )
iW1 = 1 * 1 + 0 * -1 = 1,
1 * -1 + 0 * 1 = -1 { 1 - 1}
-x
h = F(i W1) = 1 / (1 + e ) {F(1) , F (- 1) } = { 0. 73, 0.27}
hW2 =
0.27}
0.73 * -1 + 0.27 * 0 = -0.73,
0.73 * 0 + 0.27 * -1 = -0.27 { -0.73 -
o = F(i W2) = 1 / (1 + e -x ) {F(-.73) , F (-.27) }
= { 0. 675, 0.567}
d1 = 0.7( 1 -0.7)(0.7 - 1) = 0.7 (0.3)(-0.3) = -0.063
d2 = 0.6(1 - 0.6)(0.6 - 1) = 0.6(0.4)(-0.4) = -0.096
e = h(1 - h)W2d
e1 = h1(1-h1)+ h2(1-h2) W2 d =
(h1, h2)(1 - h1)( W11 W12)(D1) =
(1 - h2)(W21 W22)(D2)
e1 = (h1(1-h1)+ h2(1-h2)) W11 D1 +W12D2
e2 =( h1(1-h1)+ h2(1-h2)) W21 D1 +W22D2
e1 = (0.73(1-0.73)+ 0.27(1-0.27))( -1* -0.063 +0*-0.096)
e2 =( 0.73(1-0.73)+ 0.27(1-0.27)) (0 *-0.063 +-1*-0.096)
e1 = (0.73(0.27)+ 0.27(0.73))( 0.063)
e2 =( 0.73(0.27)+ 0.27(0.73)) (0.096)
E1 = 0.3942 * 0.063 = 0.247
E2 = 0.3942 * 0.096 = 0.038
 W2t =  hd +   W2t-1
hd = (h1) (d1 d2) = (h1d1 h1d2)
(h2)
(h2d1 h2d2)
where  = 1
1 *hd = (0.73) (-0.063 -0.096) = (0.73*-0.063 0.73*-0.096) = (-0.046 -0.107)
(0.27)
(0.27*-0.063 0.27*-0.096) (-0.017 -0.026 )