NEURAL NETWORKS:
A SOFTWARE ENGINEERING PERSPECTIVE
A Thesis
Presented to the
Faculty of the
School of Computer Science
Kennedy-Western University
In Partial Fulfillment
of the Requirements for the Degree of
Doctor of Philosophy in
Computer Science
by
David C. Waterman
Leominster, Massachusetts
1
Abstract of Dissertation
NEURAL NETWORKS:
A SOFTWARE ENGINEERING PERSPECTIVE
by
David C. Waterman
Kennedy-Western University
THE PROBLEM
The composition of an associative memory is a process or device for
learning or memorizing an association between an input pattern and an
output pattern (Looney, 1997). Traditional statistical methods employed to
process pattern associations on large data sets, due to their serial nature,
are complex and computationally expensive. These methods are subject to
limitations in their noise rejection capabilities and are severely limited in
their ability to handle local memory corruption and device failures.
Parallel models of distributed memories, such as Neural Networks,
are modeled after biological memory systems. Neural networks are
extremely powerful parallel processing systems. They can be trained to
classify, match, or complete complex patterns. Neural Networks are used in
signal detection, adaptive signal filtering, array signal processing, and signal
compression. Neural Networks have been employed to solve constrained
optimization problems and time series simulation and prediction.
Neural Networks are modeled as dynamic parallel systems and their
analysis is deeply rooted in complex, numerous, and diverse fields of
mathematics more suitable to the Mathematician or Physicist than the
Software Engineer. Software Engineers schooled in the algebra of
algorithms and data structures will avoid using complex theoretical or
mathematical models. The typical software engineer is more interested in
function over form and implementation detail rather then mathematical
theory. The goal of this paper is to make these powerful Neural Network
systems available to the Software Engineer, to bridge the gap between the
complex theoretical world of mathematical modeling to the practical
implementation of Software Engineering.
METHOD
Neural Network Architectures, that is, their topologies, learning
methods, recall methods, and activation functions were selected for
analysis and their mathematical and operational characteristics were
developed programmatically. These Neural Network building blocks were
developed using the ANSI C programming language and synthesized on a
personal computer using off-the-shelf hardware and software components.
The review of the literature focused on the fundamentals necessary to build
neural network processing elements, topologies, learning and recall
methods, and activation functions. The goal of this study was the
development of a practical software implementation for the complex
mathematical properties of Neural Networks.
FINDINGS
There are numerous Neural Network topologies of varying
complexity. The programmatical building blocks of Neural Networks
developed in Chapter 2 were assembled as a system and run as a
software simulation. Based on their similarities and ease of synthesis, the
Neural Network architectures chosen for simulation were the Discrete
Hopfield Model and the Discrete Bi-Directional Associative Memory
Model. The key issues revealed were the computational complexity,
spurious attractor states, crosstalk, activation function anomalies, and
storage capacity limitations of the neural models selected for simulation.
An area for further investigation is the construction of neural
memories in VLSI hardware and their implementation on the next
generation massively parallel hardware systems. An additional area of
further research is the aid and impact of Neural Networks in the fields of
Computational and Artificial Intelligence.
TABLE OF CONTENTS
METHOD
3
FINDINGS
4
CHAPTER 1
1
INTRODUCTION/IMPORTANCE OF THE STUDY
1
PURPOSE OF THE STUDY
ERROR! BOOKMARK NOT DEFINED.
OVERVIEW OF THE STUDY
ERROR! BOOKMARK NOT DEFINED.
PROBLEM STATEMENT
ERROR! BOOKMARK NOT DEFINED.
RATIONALE OF THE STUDY/DEVELOPING AN HYPOTHESIS ERROR! BOOKMARK NOT
DEFINED.
SCOPE OF THE STUDY
ERROR! BOOKMARK NOT DEFINED.
DEFINITION OF TERMS
1
REVIEW OF RELATED LITERATURE
2
CHAPTER 2
2
BACKGROUND AND HISTORICAL PERSPECTIVE
2
NEURAL NETWORKS PROPERTIES
2
SYNTHETIC NEURAL DEVICES
ERROR! BOOKMARK NOT DEFINED.
MCCULLOCH-PITTS NEURAL LINEAR DISCRIMINATION
ERROR! BOOKMARK NOT
DEFINED.
COMPONENTS AND TERMINOLOGY
ERROR! BOOKMARK NOT DEFINED.
CONSTITUENT ELEMENTS
ERROR! BOOKMARK NOT DEFINED.
NEURAL NETWORK COMPONENTS
ERROR! BOOKMARK NOT DEFINED.
NEURAL NETWORK ACTIVATION FUNCTIONS
ERROR! BOOKMARK NOT DEFINED.
LINEAR THRESHOLD FUNCTION
ERROR! BOOKMARK NOT DEFINED.
STEP FUNCTION
3
RAMP FUNCTION
ERROR! BOOKMARK NOT DEFINED.
BINARY SIGMOID
ERROR! BOOKMARK NOT DEFINED.
BIPOLAR SIGMOID (HYPERBOLIC TANGENT)
ERROR! BOOKMARK NOT DEFINED.
GAUSSIAN FUNCTION
SUMMARY OF ACTIVATION FUNCTIONS
TOPOLOGY
FEEDFORWARD, FEEDBACK
5
6
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
LAYERS
7
COMMON NEURAL NETWORK TOPOLOGIES
DEFINED.
ERROR! BOOKMARK NOT
INSTAR, OUTSTAR
SINGLE LAYER
MULTI-LAYER
RANDOMLY CONNECTED NETWORKS
SUMMARY OF TOPOLOGIES
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
NEURAL NETWORK LEARNING
ERROR! BOOKMARK NOT DEFINED.
HEBB’S CONTRIBUTIONS ARE IMPLEMENTED IN MOST NEURAL NETWORK
ARCHITECTURES TODAY.
ERROR! BOOKMARK NOT DEFINED.
NOTATION
ERROR! BOOKMARK NOT DEFINED.
NEURAL NETWORK LEARNING MECHANICS
DEFINED.
SUPERVISED VS. UNSUPERVISED
STRUCTURAL VS. TEMPORAL
ON-LINE VS. OFF-LINE
OFF-LINE
ON-LINE
NEURAL LEARNING AND RECALL
ERROR! BOOKMARK NOT
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
WEIGHT ADJUSTMENT
LEARNING ALGORITHMS
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
GENERALIZED HEBBIAN LEARNING
ERROR! BOOKMARK NOT DEFINED.
DISCRETE HOPFIELD LEARNING
ERROR! BOOKMARK NOT DEFINED.
DISCRETE BI-DIRECTIONAL ASSOCIATIVE (BAM) LEARNING ERROR! BOOKMARK NOT
DEFINED.
RECALL ALGORITHMS
ERROR! BOOKMARK NOT DEFINED.
DISCRETE HOPFIELD RECALL
ERROR! BOOKMARK NOT DEFINED.
DISCRETE BI-DIRECTIONAL ASSOCIATIVE (BAM) RECALL
BOOKMARK NOT DEFINED.
SUMMARY OF LEARNING AND RECALL
DEFINED.
ERROR!
ERROR! BOOKMARK NOT
SUMMARY OF CHAPTER 2
ERROR! BOOKMARK NOT DEFINED.
METHODOLOGY
ERROR! BOOKMARK NOT DEFINED.
CHAPTER 3
ERROR! BOOKMARK NOT DEFINED.
APPROACH
ERROR! BOOKMARK NOT DEFINED.
TERMINOLOGY AND DATA REPRESENTATION
DEFINED.
DATABASE OF STUDY
VALIDITY OF DATA
VECTOR SPACE
ERROR! BOOKMARK NOT
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
SCALAR PRODUCT
MATRIX PRODUCT
MATRIX TRANSPOSE
ORTHOGONALITY
LINEAR SEPARABILITY
CROSS-TALK
CAPACITY
LIMITATIONS OF DATA
ASSOCIATIVE NEURAL NETWORKS
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ASSOCIATIVE TASKS
ADDRESSABLE MEMORIES
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ADDRESS-ADDRESSABLE MEMORY
CONTENT-ADDRESSABLE MEMORY
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
AUTO-ASSOCIATIVE NEURAL NETWORKS
DEFINED.
ERROR! BOOKMARK NOT
DISCRETE HOPFIELD NEURAL NETWORK
ERROR! BOOKMARK NOT DEFINED.
CONSTRUCTION OF THE DISCRETE HOPFIELD NEURAL NETWORKERROR! BOOKMARK
NOT DEFINED.
OPERATION OF THE DISCRETE HOPFIELD NEURAL NETWORK
BOOKMARK NOT DEFINED.
ERROR!
STORAGE PHASE OF THE DISCRETE HOPFIELD NEURAL NETWORK
ERROR!
BOOKMARK NOT DEFINED.
PATTERN INPUT PHASE OF THE DISCRETE HOPFIELD NETWORK ERROR! BOOKMARK
NOT DEFINED.
RECALL PHASE OF THE DISCRETE HOPFIELD NEURAL NETWORK ERROR! BOOKMARK
NOT DEFINED.
HOPFIELD SIGNUM FUNCTION IDIOSYNCRASIES
ERROR! BOOKMARK NOT DEFINED.
HETERO-ASSOCIATIVE NEURAL NETWORKS
DEFINED.
ERROR! BOOKMARK NOT
DISCRETE BI-DIRECTIONAL ASSOCIATIVE MEMORY (BAM)
BOOKMARK NOT DEFINED.
CONSTRUCTION OF THE DISCRETE BAM
OPERATION OF THE DISCRETE BAM
ERROR!
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
PATTERN INPUT PHASE OF THE DISCRETE HOPFIELD NETWORK ERROR! BOOKMARK
NOT DEFINED.
RECALL PHASE OF THE DISCRETE BAM
ERROR! BOOKMARK NOT DEFINED.
BAM ACTIVATION FUNCTION
ERROR! BOOKMARK NOT DEFINED.
SOURCE CODE FOR DISCRETE HOPFIELD SIMULATOR
BOOKMARK NOT DEFINED.
SOURCE CODE FOR DISCRETE BAM SIMULATOR
NOT DEFINED.
ERROR!
ERROR! BOOKMARK
SUMMARY OF CHAPTER 3
ERROR! BOOKMARK NOT DEFINED.
CHAPTER 4
ERROR! BOOKMARK NOT DEFINED.
DATA ANALYSIS
ERROR! BOOKMARK NOT DEFINED.
DEVELOPMENT SYSTEM DESCRIPTION
DEFINED.
HARDWARE
SOFTWARE
NEURAL DATA REPRESENTATION
ERROR! BOOKMARK NOT
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
TRAINING SET DATA REPRESENTATION
DEFINED.
ERROR! BOOKMARK NOT
AUTO-ASSOCIATIVE TRAINING SET (GRAPHICAL) ERROR! BOOKMARK NOT DEFINED.
AUTO-ASSOCIATIVE TRAINING SET (VECTOR)
ERROR! BOOKMARK NOT DEFINED.
AUTO-ASSOCIATIVE TRAINING SET (ANSI C)
ERROR! BOOKMARK NOT DEFINED.
HETERO-ASSOCIATIVE TRAINING SET (GRAPHICAL)
ERROR! BOOKMARK NOT
DEFINED.
HETERO-ASSOCIATIVE TRAINING SET (VECTOR) ERROR! BOOKMARK NOT DEFINED.
HETERO-ASSOCIATIVE TRAINING SET (ANSI C)
ERROR! BOOKMARK NOT DEFINED.
HETERO-ASSOCIATIVE INPUT PATTERNS
ERROR! BOOKMARK NOT DEFINED.
HETERO-ASSOCIATIVE OUTPUT PATTERNS
ERROR! BOOKMARK NOT DEFINED.
ANALYSIS OF THE DISCRETE HOPFIELD NEURAL NETWORK
BOOKMARK NOT DEFINED.
ERROR!
ANALYSIS PROTOCOL
ERROR! BOOKMARK NOT DEFINED.
HOPFIELD NEURAL NETWORK LEARNING CAPABILITY
ERROR! BOOKMARK NOT
DEFINED.
HOPFIELD NEURAL NETWORK NOISE IMMUNITY PROTOCOL ERROR! BOOKMARK NOT
DEFINED.
HOPFIELD NEURAL NETWORK CAPACITY PROTOCOL
ERROR! BOOKMARK NOT
DEFINED.
HOPFIELD NEURAL NETWORK ANALYSIS RESULTS ERROR! BOOKMARK
NOT DEFINED.
TEST HOPFIELD FOR INDIVIDUAL TEST PATTERN RECALL
BOOKMARK NOT DEFINED.
ERROR!
TEST PATTERN ‘0’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘1’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘2’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘3’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘4’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘6’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘9’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘X’:
ERROR! BOOKMARK NOT DEFINED.
TEST HOPFIELD FOR INDIVIDUAL TEST PATTERN RECALL SUMMARY
ERROR!
BOOKMARK NOT DEFINED.
TEST HOPFIELD FOR TEST PATTERN NOISE IMMUNITY
BOOKMARK NOT DEFINED.
STABLE POINTS IN PATTERN STATE SPACE
ERROR!
ERROR! BOOKMARK NOT DEFINED.
TEST HOPFIELD FOR TEST PATTERN NOISE IMMUNITY SUMMARY
ERROR! BOOKMARK NOT DEFINED.
TEST HOPFIELD FOR CAPACITY
ERROR! BOOKMARK NOT DEFINED.
ANALYSIS OF THE DISCRETE BI-DIRECTIONAL ASSOCIATIVE MEMORY
ERROR! BOOKMARK NOT DEFINED.
ANALYSIS PROTOCOL
ERROR! BOOKMARK NOT DEFINED.
BAM NEURAL NETWORK NOISE IMMUNITY PROTOCOL
ERROR! BOOKMARK NOT
DEFINED.
BAM NEURAL NETWORK CAPACITY PROTOCOL ERROR! BOOKMARK NOT DEFINED.
BAM NEURAL NETWORK ANALYSIS RESULTS
DEFINED.
ERROR! BOOKMARK NOT
TEST BAM FOR INDIVIDUAL TEST PATTERN RECALLERROR! BOOKMARK
NOT DEFINED.
TEST PATTERN ‘0’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘1’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘2’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘3’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘4’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘6’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘9’:
ERROR! BOOKMARK NOT DEFINED.
TEST PATTERN ‘X’:
ERROR! BOOKMARK NOT DEFINED.
TEST BAM FOR INDIVIDUAL TEST PATTERN RECALL SUMMARY ERROR! BOOKMARK
NOT DEFINED.
TEST BAM FOR TEST PATTERN NOISE IMMUNITY
NOT DEFINED.
ERROR! BOOKMARK
TEST BAM FOR TEST PATTERN NOISE IMMUNITY SUMMARY
BOOKMARK NOT DEFINED.
TEST BAM FOR CAPACITY
TEST BAM FOR TEST CAPACITY SUMMARY
ERROR!
ERROR! BOOKMARK NOT DEFINED.
ERROR! BOOKMARK NOT DEFINED.
SUMMARY OF CHAPTER 4
ERROR! BOOKMARK NOT DEFINED.
SUMMARY
ERROR! BOOKMARK NOT DEFINED.
CHAPTER 5
ERROR! BOOKMARK NOT DEFINED.
BIBLIOGRAPHY
ERROR! BOOKMARK NOT DEFINED.
Page 1
Neural Networks: A Software Engineering Perspective
Chapter 1
Introduction/Importance of the Study
Definition of Terms
Page 2
Neural Networks: A Software Engineering Perspective
Review of Related Literature
Chapter 2
Background and Historical Perspective
Neural Networks Properties
Page 3
Step Function
Page 4
Page 5
Gaussian Function
Page 6
Summary of Activation Functions
Page 7
Layers
Page 8
Page 9
Page 10
Page 11
Hopfield, J.J., & D. Tank. “Neural Computation of decisions in
Optimization Problems.” Biological Cybernetics, 1985.
Kohonen, T. “A Simple Paradigm for the Self-Organized Formation of
Structured Feature Maps”, in Competition and Cooperation in
Neural Nets, Lecture Notes in Biomathematics. Springer-Verlag,
1982.
Kohonen, T. Content-Addressable Memories, 2nd Edition. Springer-Verlag,
1987.
Kohonen, T., “The Self-organizing Map”, Proc. IEEE 78(9), 1990.
Kosko, B. “Bi-directional Associative Memories”, IEEE Trans. Syst. Man
Cybern, 1987.
Kosko, B. Neural Networks and Fuzzy Systems, A Dynamical Approach to
Machine Intelligence. Prentice Hall, 1992.
Landau, L., & J. Taylor. Concepts for Neural Networks. Springer-Verlag,
1997.
Leondes, C. Algorithms and Architectures. Neural Network Systems
Techniques and Applications. Academic Press, 1998.
Looney, Carl G. Pattern Recognition Using Neural Networks. Oxford
University Press, 1997.
McEliece, R.J., E.C. Posner, E.R. Rodemich, S.S. Venkatesh. “The
Capacity of the Hopfield Associative Memory.” IEEE Transactions
on Information Theory, vol. IT-33, 461-482, 1987.
McCulloch, W.S., & W. Pitts. A Logical Calculus of the Ideas Immanent in
Nervous Activity. Bulletin of Mathematical Biophysics, 1943.
Minsky M.L. & S.A. Papert. Perceptrons. MIT Press, 1969.
Skapura D., & J Freeman. Neural Networks, Algorithms, Applications, and
Programming Techniques. Computation and Neural Systems
Series. Addison-Wesley, 1992.
Page 12
Skapura, D. Building Neural Networks. Addison-Wesley, 1996.
Swingler, K. Applying Neural Networks, A Practical Guide. Academic
Press, 1996.
White, H. Learning in Neural Networks: A Statistical Perspective. Neural
Computation (publication), 1989.
Zurada, J.M. Introduction to Artificial Neural Systems. West Publishing,
1992.