Artificial Neural Networks
ANN application development
Neural Network Applications Development
Neural Network Implementations
Neural Network Development Methodology
-
The Concept Phase
The Design Phase
Implementation
Maintenance
1
Artificial Neural Networks
ANN application development
Neural Network Implementations
Three possible practical implementations:
1.
A software simulation program running on a digital
computer.
2.
A hardware emulator connected to a host computer - the
so-called neurocomputer.
3.
A true electronic or optical neural network.
Software Simulations
Cheapest and fastest implementation method.
Replicates temporal behaviour of a network by numerically
solving the system's equations for successive instants of time.
Written in most hi-level languages, C most popular. Trend
toward object-oriented languages.
Drawbacks:
Writing simulators can be time consuming.
Solution - prewritten shells (commercial neural net packages)
with convenient user interfaces. User must specify type of
architecture and interconnections, number of layers and nodes.
Simulations are slow. Time and resource requirements
become unacceptable beyond a few hundred nodes.
Inherent uncertainty of modelling - some combinations of
parameters might produce nonsense. Too sensitive to initial
conditions.
Inherent error of numerical computation, eg, truncation
errors, approximation of differential equations with difference
equations.
2
Artificial Neural Networks
ANN application development
Neurocomputers
A dedicated special-purpose digital computer optimised to
perform operations common in neural network simulation.
Acts as a coprocessor to a host computer and controlled by
program run on the host.
Tens to thousands of times faster than simulators. Systems
available with over 10 million IPS (Interconnect updates Per
Second). Can emulate neural networks with several million
neurones. Speed still limited by the need to maintain arrays in
memory with thousands or millions of elements.
True Networks in Hardware
Neurons fabricated on silicon chips. Commercially available.
Limitation of a few thousand neurons per chip. Chips connected
in parallel to achieve larger networks. Problems with
interconnection and interference. Simple synapses are resistors
which represent fixed-valued weights. Work progressing on
modifiable synapses.
Hybrid electro-optical or fully optical neural networks are
currently undergoing research and development and are
expected to be available by the turn of the century.
3
Artificial Neural Networks
ANN application development
Neural Network Development Methodology
Aims to add structure and organisation to neural network
applications development to reduce cost, increase accuracy,
consistency, user confidence and friendliness.
The Concept Phase
Involves
1. validating proposed application, and
2. selecting a neural paradigm.
Some problem characteristics suitable for neural network
application:
-
Data intensive
Multiple interacting parameters
Incomplete, erroneous data
solution function unknown or expensive
requires flexibility, generalisation, fault-tolerance,
speed
Examples: Character recognition, forecasting, signal processing,
Problems not suitable:
Mathematically accurate and precise applications,
solution involves deduction and step-wise logic,
involves reporting.
Examples: resource management eg, inventory
accounts, sales data analysis
4
Artificial Neural Networks
ANN application development
Selecting a neural paradigm
Based on comparison of application requirements to neuralparadigm capabilities. Issues to be considered:
Neural network size. Problem should be scaled to a size
suitable for neural computing capabilities. Statistical neural
network paradigms such as Boltzmann work best for smaller
networks (around a few hundred) compared with nonstatistical
paradigms such as backprop.
Required output type. Four common interpretations of output:
classification, pattern, real number, and optimisation.
Associative memory classification.
Autoassociative, eg, character recognition, signal reconstruction.
Heteroassociative, eg, process monitoring, financial trend
analysis.
Training Method.
Supervised (must have correct output),
Unsupervised (less data-requirement and training time)
Time Constraints. Training time (significantly longer) and
Execution time.
After the proposed application is validated and the neural
network paradigm selected, its technical and economic
feasibility assessment should be carried out to complete the
concept phase.
5
Artificial Neural Networks
ANN application development
The Design Phase
Specifies initial values and conditions at the node, network and
training levels.
The node level. Decisions to be made:
Types of input binary, bipolar, discrete, continuous.
Transfer function sigmoid (continuous inputs),
hard limiting (binary, bipolar inputs)
lookup tables used for speeding up calculations
The network level.
Number, size and connectivity of layers.
Number:
One or two layers - Hopfield, Kohonen, ART,
One or more hidden layers - Backprop, Neocognitron
Size (number of nodes in) of layers:
Input layer determined by number of data sources (input
vector components)
Hidden layer - needs experimentation. Too few - incorrect
mapping, too many - inadequate generalisation.
Output layer - determined by number of classes or objects to
be output.
6
Artificial Neural Networks
ANN application development
Connectivity
Feedforward (no feedback, no intralayer connections, eg,
backprop nets)
Feedback (eg, ART)
Intralayer connections ( eg, Hopfield - fully connected,
Kohonen)
Other design questions to be addressed:
-
Setting of parameters for learning phase
-
Possible addition of noise to speed up convergence in
gradient descent
-
Possible combination of different neural network
paradigms to meet application objectives better.
References
1.
Caudill & Butler, Naturally Intelligent Systems, MIT Press
1989, pp 227-240.
2.
Bailey, D., & Thompson, D., How to Develop NeuralNetwork Applications, AI Expert, June 1990, pp. 38-47.
7
Artificial Neural Networks
ANN application development
The Implementation phase
Objective:
A functional neural network based on the most accurate,
consistent and robust model possible.
Gathering the training set
Four aspects Type of data to collect :
depends on learning algorithm, type of input and output
nodes, and nature of the problem. Data should be
representative including routine, unusual and boundarycondition cases; non-ambiguous and not too erroneous.
Amount of data to gather:
Increasing data amount increases training time but may
help earlier convergence. May be limited by time and other
resources availability. Quality more important than
quantity.
Collection of data:
May be tedious if multiple data sources exist. Potential
sources include - historical records, instrument readings,
simulation results, test data.
Preparation of data:
May involve - preprocessing, normalising, converting into
binary vector.
Selecting the developing environment
Two aspects - Hardware and software.
Hardware: Requirements based on speed of operation, memory
and storage capacity, software availability, cost, compatibility
etc.
Most popular platforms - workstations and PC's (with
accelerator board option); mainframes, parallel processors also
used.
8
Artificial Neural Networks
ANN application development
Software: Two options Custom-coded neural paradigms - maximum flexibility
Commercial development packages - easy to use, selection
based on :
neural paradigms implemented
user programmable paradigms
speed in training and recall
transportability
vendor support
extensibility
price
Implementing the neural network
Common steps involved:
1. Select appropriate neural paradigm
2. Set network size
3. Determine the learning algorithm
4. Create any screen displays
5. Load the training set
6. Initialise the network
7. Determine the halting criteria
8. Begin training the network
9. Monitor the training process.
Possible problems during training - stabilising in a local minima,
overtraining. May require resetting the learning parameters
and reinitialising, if addition of small amount of noise does
not help.
Training results may be affected by the method of presenting
data set to the network. Some commercial packages use a
random rather than sequential order; others buffer weight
updates from individual data sets before combining them.
To ensure optimal results, several variations of a neural network
may be trained and each tested for accuracy. Adjustments
may be made by varying the layers sizes and fine tuning the
learning parameters
9
Artificial Neural Networks
ANN application development
Testing and debugging the network
Testing can be done by:
1. Observing operational behaviour of the net.
2. Analysing actual weights
3. Study of network behaviour under specific conditions
1.
Testing usually done by treating the network as a black
box and evaluating its response to a series of test cases.
Thresholds for acceptance of network responses must be
predetermined.
Test data should contain training cases as well as new cases.
Routine, difficult as well as boundary condition cases should be
tried.
2.
The weights entering and exiting nodes are analysed. Any
nodes without significant connections (>0.1) is a candidate for
deletion. Similarly, any significantly large weight might indicate
a problem.
3.
Specific nodes may be strongly activated and its function
in network behaviour observed (is the input output relationship
reasonable?).
In case of significant errors detected in testing, debugging would
involve examining
- the training cases (representativeness, accuracy,
number);
- learning algorithm constants;
- neural network architecture, node characteristics, and
connectivity;
- training set-network interface, user-network interface
10
Artificial Neural Networks
ANN application development
The Maintenance Phase
Consists of placing the neural network in an operational
environment with possible integration, periodic performance
evaluation, and maintenance.
Operational environment
NN
ES
DB
Stat
NN
DISTRIBUTED
STAND-ALONE
Expert System
Signal
NN
ES
PREPROCESSOR
NN
NN
EMBEDDED
DB
NN
POSTPROCESSOR
Fig. 11.1 Operational environments for neural networks
Although often designed as stand-alone systems, some neural
network systems are integrated with other information systems
using loose-coupling, tight-coupling or full integration.
In loosely-coupled systems, the neural network is a
preprocessor, a postprocessor or a distributed component
communicating by passing data files.
11
Artificial Neural Networks
ANN application development
In a tightly-coupled system, the neural network is fully
integrated and works in close cooperation with the rest of the
system for problem solving, eg, neural networks embedded in
expert systems.
Neuro-symbolic computing utilises the complementary nature of
computing in neural networks (numerical) and expert systems
(symbolic).
Neuro-fuzzy systems combine neural networks with fuzzy logic.
Another type of hybrid intelligent systems combine ANNs with
genetic algorithm methodology (See "Hybrid Intelligent
Systems" by L. Medsker, Kluwe Academic Press, Boston 1995)
System evaluation
Continual evaluation is necessary to ensure satisfactory
performance in solving dynamic problems, and to check
damaged or retrained networks.
Autonomous systems like the ART should be monitored against
biasing with rare situations.
Evaluation can be carried out by reusing original test procedures
with current data.
12
Artificial Neural Networks
ANN application development
Maintenance
Involves modification due to decreasing accuracy or
enhancements.
Two categories:
Data modification
- carried out in two steps. First, the training data must be
modified or replaced; second, the network retrained and reevaluated.
Software changes
- include changes in interfaces, cooperating programs or in the
structure of the network. If the network is changed, part of the
design and most of the implementation phase may have to be
repeated.
Backup copies should be used for maintenance and research.
13
Artificial Neural Networks
ANN application development
Artificial Neural Networks - Looking Ahead
The promise of neural nets:
 Ability to solve problems resistant to algorithmic solutions on
digital computers, eg, pattern recognition, optimisation.
 Potential as associative memory with ability to handle
incomplete, erroneous data
 Capability to generalise
 Speed of operation due to inherent parallelism
 Fault-tolerance due to distributed nature
 An avenue for extending understanding of biological systems
Widespread application held back by limitations of present day
hardware technology.
Major breakthroughs in fabrication technology, optical and
opto-electronic technology expected within a decade(?).
Biological (natural) neural network systems using genetic
engineering by the middle(?) of the century.
Artificial neural nets - a substitute for conventional technology?
- Unlikely to happen. More likely to appear in hybrid intelligent
systems complementing expert systems, fuzzy logic and genetic
algorithmic computing.
14