Neural network
ARTIFICIAL NEURAL NETWORKS
Also referred to as connectionist architectures, parallel distributed
processing, and neuromorphic systems, an artificial neural network (ANN)
is an information-processing paradigm inspired by the way the densely
interconnected, parallel structure of the mammalian brain processes
information. Artificial neural networks are collections of mathematical
models that emulate some of the observed properties of biological nervous
systems and draw on the analogies of adaptive biological learning. The key
element of the ANN paradigm is the novel structure of the information
processing system. It is composed of a large number of highly
interconnected processing elements that are analogous to neurons and are
tied together with weighted connections that are analogous to synapses.
Learning in biological systems involves adjustments to the synaptic
connections that exist between the neurons. This is true of ANNs as well.
Learning typically occurs by example through training, or exposure to a
truthed set of input/output data where the training algorithm iteratively
adjusts the connection weights (synapses). These connection weights store
the knowledge necessary to solve specific problems.
Although ANNs have been around since the late 1950's, it wasn't until the
mid-1980's that algorithms became sophisticated enough for general
applications. Today ANNs are being applied to an increasing number of
real- world problems of considerable complexity. They are good pattern
recognition engines and robust classifiers, with the ability to generalize in
making decisions about imprecise input data. They offer ideal solutions to
a variety of classification problems such as speech, character and signal
recognition, as well as functional prediction and system modeling where
the physical processes are not understood or are highly complex. ANNs
may also be applied to control problems, where the input variables are
measurements used to drive an output actuator, and the network learns the
control function. The advantage of ANNs lies in their resilience against
distortions in the input data and their capability of learning. They are often
good at solving problems that are too complex for conventional
technologies (e.g., problems that do not have an algorithmic solution or for
which an algorithmic solution is too complex to be found) and are often
well suited to problems that people are good at solving, but for which
traditional methods are not.
There are multitudes of different types of ANNs. Some of the more popular
include the multilayer perceptron which is generally trained with the
backpropagation of error algorithm, learning vector quantization, radial
basis function, Hopfield, and Kohonen, to name a few. Some ANNs are
classified as feedforward while others are recurrent (i.e., implement
feedback) depending on how data is processed through the network.
Another way of classifying ANN types is by their method of learning (or
training), as some ANNs employ supervised training while others are
referred to as unsupervised or self-organizing. Supervised training is
analogous to a student guided by an instructor. Unsupervised algorithms
essentially perform clustering of the data into similar groups based on the
measured attributes or features serving as inputs to the algorithms. This is
analogous to a student who derives the lesson totally on his or her own.
ANNs can be implemented in software or in specialized hardware.
1. The Perceptron
A Learning Program
For decades there have been attempts to create computer programs that can
learn like people - Artificial Intelligence.
For example, how do you teach a child to recognize what a chair is? You show him
examples telling him "This is a chair ; That one is not a chair" until the child learns
the concept of what a chair is. In this stage, the child can look at the examples we
have shown him and answer correctly to the question "Is this object a chair?".
Furthermore, if we show to the child new objects, that he didn't see before, we
could expect him to recognize correctly whether the new object is a chair or not,
providing that we've given him enough positive and negative examples. This is
exactly the idea behind the perceptron.
The Perceptron
The perceptron is a program that learn concepts, i.e. it can learn to respond
with True (1) or False (0) for inputs we present to it, by repeatedly "studying"
examples presented to it.
The Perceptron is a single layer neural network whose weights and biases could be
trained to produce a correct target vector when presented with the corresponding
input vector. The training technique used is called the perceptron learning rule. The
perceptron generated great interest due to its ability to generalize from its training
vectors and work with randomly distributed connections. Perceptrons are especially
suited for simple problems in pattern classification.
Our perceptron network consists of a single neuron connected to two inputs
through a set of 2 weights, with an additional bias input.
The perceptron calculates its output using the following equation:
P * W + b > 0
where P is the input vector presented to the network, W is the vector of
weights and b is the bias.
The Learning Rule
The perceptron is trained to respond to each input vector with a corresponding
target output of either 0 or 1. The learning rule has been proven to converge on
a solution in finite time if a solution exists.
The learning rule can be summarized in the following two equations:
For all i:
W(i) = W(i) + [ T - A ] * P(i)
b
= b + [ T - A ]
where W is the vector of weights, P is the input vector presented to the
network, T is the correct result that the neuron should have shown, A is the
actual output of the neuron, and b is the bias.
Training
Vectors from a training set are presented to the network one after another. If
the network's output is correct, no change is made. Otherwise, the weights and
biases are updated using the perceptron learning rule. An entire pass through
all of the input training vectors is called an epoch. When such an entire pass of
the training set has occured without error, training is complete. At this time
any input training vector may be presented to the network and it will respond
with the correct output vector. If a vector P not in the training set is presented
to the network, the network will tend to exhibit generalization by responding
with an output similar to target vectors for input vectors close to the
previously unseen input vector P.
Limitations
Perceptron networks have several limitations. First, the output values of a
perceptron can take on only one of two values (True or False). Second,
perceptrons can only classify linearly separable sets of vectors. If a straight
line or plane can be drawn to seperate the input vectors into their correct
categories, the input vectors are linearly separable and the perceptron will find
the solution. If the vectors are not linearly separable learning will never reach
a point where all vectors are classified properly.
The most famous example of the perceptron's inability to solve problems with
linearly nonseparable vectors is the boolean exclusive-or problem.
Our Implementation
We implemented a single neuron perceptron with 2 inputs. The input for the
neuron can be taken from a graphic user interface, by clicking on points in a
board. A click with the left mouse button generates a '+' sign on the board,
marking that it's a point where the perceptron should respond with 'True'. A
click with the right mouse button generates a '-' sign on the board, marking
that it's a point where the perceptron should respond with 'False'. When
enough points have been entered, the user can click on 'Start', which will
introduce these points as inputs to the perceptron, have it learn these input
vectors and show a line which corresponds to the linear division of the plane
into regions of opposite neuron response.
Here's an example of a screen shot:
Examples
Here are more examples of screen shots. The first two show succesful runs of
the perceptrons, the third is an example of how the perceptron fails to classify
a set of non linearly separable vectors.

Example 1 ( GIF, 10K )

Example 2 ( GIF, 10K )
Example 3 ( GIF, 10K )

Source Code
Here is the source code. In order to be able to run the program, save all these
files in a directory, type 'make', and use 'prcpt' to start the system. The program
should run under all platforms which support Tk, though we tested it for
UNIX SunOS only.

prcpt - The Tk interface (8K).

perceptron.c - The actual Neural Network, C code (7K).
readme - The readme file (very much like this file) (6K).
makefile - The (trivial) makefile (<1K).


You can also grab all these files in one tarred gzipped file (6K) .
Note: This code is free and is in the public domain. You may copy, use, and
change it as much as you like. However, the writers take no responsibility for the
results of using this program.
See Also

NNUGA - Neural Network Using Genetic Algorithms
2.Backpropagation
XOR applet is a little example that implements a 3-input XOR gate
with a 3 layered neuronal network ( BPN.class ).
Wait until a window appear in your screen. When You got the window ...

Click start learning
You'll see a 'table' performing a completely wrong 3-input XOR gate,
followed by learning performances in time. The system is learned in
10000 sessions, every 1000 sessions absolute error and relative
learning rate is printed.
Finally at the end our trained network is checked, the 'table' is reprinted
and this should be the right logic output.
Input
Output
F
F
F
F
F
F
T
T
F
T
F
T
F
T
T
F
T
F
F
T
T
F
T
F
T
T
F
F
T
T
T
T
Now You can save your well-trained network in a file (this not work on
Netscape, for security protection Filebrowser will not appear) and use it for
your private application with the help of BPN.class .
You can decide to learn the network at every moment, click Start Learning
again and the train will continue from the last state You left it.
I made this simple program just to test BPN.class and provide a tutorial for
everybody need such of little networks. XOR is just a little example, but I
used a very similar network to implement an OCR (in C) or a simple
reactive insect brain. It may be very polyuseful.
Fields
public static final double FIRE =
0.999;
public static final double NEUTRAL = 0.0;
public static final double DOWN =
-0.999;
Variables
public double absoluteError; // updated when learning.
public double inpA[];
// input activations (range: DOWN .. FIRE)
public double outA[];
// output activations (range: DOWN .. FIRE)
Constructor
public BPN(int i, int h, int o,
double eida, double theta, double elast, double momentum);
i
:= # of neurons at input layer
h
:= # of neurons at hidden layer
o
:= # of neurons at output layer
eida
:= starting learning rate
theta
:= sigmoid thresold value (range: DOWN .. FIRE)
elast
:= sigmoid elasticity ( 1.0 .. 2.0)
momentum := learning rate attenuator (range 0.1 .. 1.0,
keep it very high ~ 0.99)
Methods
public void init()
Init the network and set random values to neurons and weigths.
NOTE: init() is automatically called at construction method.
public void propagate(double[] vector)
apply vector[] at input and propagate it to get values in outA[].
Throws ArrayIndexOutOfBoundsException when vector length
don't match input layer length.
public void learnVector(double[] in, double[] out)
apply in[] at input and learn the network to produce out[].
Throws ArrayIndexOutOfBoundsException when in[] length
don't match input layer length or out[] length dont't match output
layer length.
public boolean saveNeuro(String path, String name);
Save size of the network, weigths and sigmoid parmeters
(eida, theta, elast, momentum) in a file specified by
path and file.
Return true if succesful.
public boolean loadNeuro(String path, String name);
Load and setup the network with the data in specified file.
Return true if succesful.
};
3.SOM(Self Organizatoin Map)
Some Background
Self-adaptive topological maps were initially inspired by modelling perception
systems found in mammalian brain. A perception system involves the reception of
external signals and their processing inside the nervous system. The complex
mammalian skills, such as seeing and hearing seemed to bear similarity to each
other in the way they worked. Namely, the primary characteristic of these systems
is that neighbouring neurons encode input signals which are similar to each other.
General idea of the SOM model
The Self-Organizing Map (SOM) was introduced by Teuvo Kohonen in 1982. The
SOM (also known as the Kohonen feature map) algorithm is one of the best known
artificial neural network algorithms. In contrast to many other neural networks
using supervised learning, the SOM is based on unsupervised learning.
The SOM is quite a unique kind of neural network in the sense that it constructs a
topology preserving mapping from the high-dimensional space onto map units in
such a way that relative distances between data points are preserved. The map
units, or neurons, usually form a two-dimensional regular lattice where the location
of a map unit carries semantic information. The SOM can thus serve as a clustering
tool of high-dimensional data. Because of its typical two-dimensional shape, it is
also easy to visualize.
Another important feature of the SOM is its capability to generalize. In other words,
it can interpolate between previously encountered inputs.