1
Neural Network for Dynamic Data Classification
Based on Discrete-Time Winner-Takes-All Neural Circuits
Pavlo V. Tymoshchuk1, Sergii V. Shatnyi2
Department of Computer Aided Design Systems, L’viv Polytechnic National University,
12, S. Bandera Str., L’viv, 79013, Ukraine, E-Mail: 1pautym@polynet.lviv.ua, 2sha_ser@ukr.net
Abstract – The design of functional block-diagram of neural
network classifier is proposed. The outputs are obtained by
using self-optimizing (SONN) neural network based on the
discrete-time dynamic Winner-takes-all (WTA) neural circuits
which are capable to identify the largest from N input
features. Implementation prospects of the SONN in an up-todate digital hardware and its application for classification
problem solving of dynamic features are outlined. The network
combines such properties as high speed and precision of data
processing.
Кеу words – neural network, SONN, WTA, classification,
descrete time.
I. Introduction
As known, the self-optimizing neural network (SONN)
is a neural network classifier based on a training data
analysis which quickly estimates values of individual real,
integer or binary input features. The SONN topology
consists of the following three types of neurons that fulfill
an input data transformation from real input vectors into a
final classified outputs: discriminatively-factorization
neurons,
aggregation-strengthening
neurons
and
maximum-selection neurons. The neurons are arranged in
some number of layers. The number of layers is
dependent on training data correlations. If training data of
different classes are more correlated, then a neural
network architecture is also more complicated and it has
got more layers and neurons and vice versa [1], [2].
The SONN classification results have been compared
with the other AI methods for the Wine and Iris data from
MLRepository. The comparison results show that an error
of correct classification by SONN approach is close to
that of FSM method and method of k-nearest-neighbors.
In particular, the whole process of neural network
development and optimization and the weight
computation for classification by using SONN for 5
inputs continues 1 - 2 seconds [2]. The SONN was
successfully implemented on digital computers and
adapted to many classification and recognition problems,
in particular, to classical benchmarks, medical and
pharmacological diagnosing.
II. The problem statement
The purpose of this paper is to design the functional
block-diagram of the SONN for solving classification
problem of dynamic features. Outputs of such the network
have to be obtained based on discrete-time dynamical
Winner-takes-all (WTA) neural circuits. Prospects of the
network implementation in an up-to-date digital hardware
have to be outlined.
III. The network design for dynamic
input features
Winner-takes-all neural circuit
For every time point t(k), k = 1,2,…M different neural
network structures can be created. In each time point t(k)
corresponding network is created and optimized, and
values of its weights are computed for classification
problem solving. The process of designing the networks
for M time points of input features will require the time
T=M  , where  is a time of creating, optimization and
the network weight computing in one discrete-time point
of input feature. Thus a time of designing M networks for
M time points of input features by sequential computer
software can reach unreasonable large value for large
number of M. Moreover, in this case in order to keep
designed structure and parameters of networks for each
time point k in computer it is necessary to have large
volume of memory. For instance, according to numerical
results presented in [3] for concrete example of
classification with 5 input features the whole process of
the SONN development, optimization and weights
computation takes 1 - 2 seconds. Therefore for M=10000
the time T=10000 - 20000 seconds>2.5 – 5 hours which
can be unexeptable from practical point of view.
Furthermore, in order to solve the classification problem
of dynamic inputs by SONNs in a real time, i. e. in online
mode the inequality  <= t (k  1)  t (k ) , k=1,2,...,M-1
should be satisfied. For instance, to classify signals of
EEGs given in [4], [5] in real time it is necessary to spend
 <=78 ms that is much less than SONNs presented in [3]
can provide. Therefore, the problem of reducing the
problem solving time is important both for offline and
online classification. Such the problem can be solved
based on designing maximum-selection neurons of the
network by using Winner-takes-all (WTA) neural circuits.
WTA neural networks are known to select largest out
of a set of N inputs [6], [7]. The network outputs are
proposed to calculate by using discrete-time dynamic Kwinners-take-all (KWTA) neural circuit, where
1  K  N presented in [8] for the case K=1. It is known
that discrete-time neural networks comparatively to
continuous-time analogs demonstrate a more high
precision of signal processing, they are more reliable,
more suitable to implement in software, and can be
implemented in an up-to-date digital hardware for real
time data processing. The circuit proposed in [8] is
globally stable and convergent to WTA operation in finite
number of iterations. It is composed of N feedforward
neurons and one feedback hardlimiting neuron used to
determine the desired shift of inputs. The circuit can be
implemented in a digital hardware by summers,
“COMPUTER SCIENCE & INFORMATION TECHNOLOGIES” (CSIT’2014), 18-22 NOVEMBER 2014, LVIV, UKRAINE
2
integrator, switches and external sources of voltage or
current, which are appropriate for real time data
processing using VLSI technologies [9]. In contrast to
other comparative analogs, the circuit is capable to
process correctly any finite value distinct inputs of
arbitrary finite range, it has low computational and
hardware implementation complexity, high speed of
processing inputs, and possesses order preserving
property of inputs. The circuit does not require resetting
and corresponding supervisory circuit that additionally
simplifies the hardware and increases a speed of
processing inputs.
A functional block-diagram of the network
A functional block-diagram of the network can be
composed of three parts, in particular aggregationstrengthening neurons, discriminatively-factorization
neurons, and maximum-selection neurons [2], [3]. In
particular, a functional block-diagram of the aggregationstrengthening neurons is presented in Fig. 1. The diagram
consists of blocks of input data u training/test; classes C;
 ; summing  ; controlled switches S;
multiplication, division and summing  :  ;
multiplication and division  : ; multiplication and
summation   . The aggregation-strengthening neurons
Fig.2 Architecture of the discriminatively-factorization
neurons.
norm function
are implemented in a digital hardware using summers,
multipliers, dividers and controlled switches.
Fig.1 Architecture of the aggregation-strengthening
neurons.
A functional block-diagram of the discriminativelyfactorization neurons is presented in Fig. 2. The diagram
consists of digital comparators, AND- and NO-switches,
and constant voltage or current sources.
A functional block-diagram of the maximum-selection
neurons built based on the discrete-time dynamic KWTA
neural circuit (NC) presented in [8] for K=1 is shown in
Fig. 3, where ∑ is a block of discrete-time summation, S
is a switch. Thus, the circuit can be implemented in a
digital hardware using such digital components as
summers, controlled switches, integrator, and external
sources of voltage or current.
Fig.3 Architecture of the maximum-selection neurons
based on KWTA NC presented in [8] for K=1.
The network output has to be computed for each class
m  (1,..., M) . Therefore, in order to reduce a time of
obtaining the network outputs, parallel computation of
each output can be applied. For this purpose M discretetime dynamic KWTA NCs can be used.
The overall network can be implemented in an up-todate digital hardware using such electronic circuit
components as summers, multipliers, dividers, controlled
switches, comparators, AND- and NO-switches,
integrators, and external sources of voltage or current.
It is necessary to note that the functional blockdiagrams presented in Fig. 1 – Fig. 3 are characterized by
hardware implementation restrictions. In particular, all the
implemented blocks of the block-diagrams will have time
delays which in totality define the speed of processing
inputs by corresponding real digital network. An
expression for time delay to process one set of input
features by the network can be presented as follows:
T  T1  T2  T3 ,
(1)
T1 is a time delay of the discriminativelyfactorization neurons, T2 is a time delay of the
discriminatively-factorization neurons, T3 is a time delay
where
of the maximum-selection neurons. If more than one set
of input features should be processed, it can be done
sequentially in time. In other words, each next set of
inputs can be processed after processing the previous set
of inputs. Therefore, in order to obtain a correct operation
of the network in the case of more than one set of inputs,
a repetition period Tr of sets of input features should
meet the following inequality:
(2)
Tr  T .
“COMPUTER SCIENCE & INFORMATION TECHNOLOGIES” (CSIT’2014), 18-22 NOVEMBER 2014, LVIV, UKRAINE
3
The network can be used in the case of time-varying
input features cl (k) , l=1,2,…,N, k=1,2,… if the module
of speed change of such input features is much less than
that of their processing by the network. In other words, in
this case, the condition
(3)
dx l (k) / dt  dc l / dt
should be satisfied for each k   .
The resolution of the network is theoretically infinite
and it does not depend of values of its parameters. In
other words, the network can always classify distinct
input features correctly. Moreover, the network is capable
to operate correctly with any initial value of the state
variables 0  x l (1)  A , where A>0 [7]. Therefore, a
functioning of the designed network is independent on
initial conditions which can accept arbitrary values in a
given range [0,A]. Therefore the network will not require
periodical resetting for repetitive processing of input
features, additional supervisory circuits for resetting, and
spend additional processing time on this operation. This
allows to simplify the hardware and increase a speed of
processing input features that is important for real time
operating of the designed network.
Implementation prospects of the network
An implementation of the network can be simulated
using software. Such simulations confirm obtained
theoretical results. It is known that software
implementations of neural networks offer flexibility.
Software implementations of the network can be trained
and simulated on general-purpose sequential computers.
For reducing a time of classification of dynamic input
features the network implementation in a parallel software
using their parallel structure can be performed. However,
the processing speed of input features by the network
implemented in software can be not high enough
especially to meet demands of real time. Therefore,
microprocessors and digital signal processing can be not
suitable for parallel designs of network for real time
applications. Despite the tremendous growth in the digital
computing power of general-purpose processors neural
network hardware has been found to be promising in
many applications such as image processing, speech
synthesis and analysis, pattern recognition, high energy
physics and others.
In order to speed-up the network operation it can be
implemented in an up-to-date digital hardware.
Comparing with an analogue implementation, a digital
hardware is more computationally precise and reliable as
long as the requirements for the size and power efficiency
are not high. The digital implementation of the designed
network will have better repeatability, lower noise
sensitivity, better testability, higher flexibility, as well as
compatibility with other types of preprocessors [9].
Hardware implementations are essential for applicability
and for taking the advantage of network inherent
parallelism.
Specific-purpose
fixed
hardware
implementations (i.e. VLSI) can be dedicated to specific
network models. VLSI implementations of network is
capable to provide not only high speed in real time
applications but also compactness [10].
Therefore, to solve the problem of classification of
dynamic input featuress an implementation of the
network in an up-to-date digital hardware can be used.
For the network hardware FPGA-based implementations,
ASIC-based
implementations,
and
DSP-based
implementations can be used. Since DSP-based
implementation is sequential, it does not preserve the
parallel architecture of the network. ASIC implementation
can be used for the network hardware realization,
although it does not offer re-configurability by the user in
order to improve their performance. The FPGA
implementation achieves a comparable accuracy with the
traditional solutions based on general-purpose computers.
An FPGA as an implementation hardware combines the
reprogrammability advantage of general purpose
processors with the parallel processing and speed
advantages of customer hardware. The size and speed
evaluation of FPGA reveals its low cost in terms of logic
and memory [11]. To implement the network in a
hardware, the FPGA based reconfigurable computing
architecture is quite suitable because the parallel structure
of FPGA matches the topology of the network and offers
flexibility in reconfiguration. The architecture of the
network and training algorithms can be implemented on a
FPGA chip performing an on-line training. Such
computational characteristics of the network as
modularity and dynamic adaptation can be also realized in
FPGA hardware. Using FPGA, the network may be
implemented through parallel computing in a real-time
hand-tracking system. Due to the relatively high capacity,
high density, short design cycle, and short time to market
when using EDA tools, FPGA can be considered as the
most applicable microelectronic technology for the
network designing [12].
Neural network hardware implementation on FPGA
The software platform for the FPGA hardware
implementation allows several levels of definition of the
logic functions. One level is through an electrical
schematic diagram, a second level through hardware
description languages. These levels result in the synthesis
of logic hardware for implementation the designed neural
network.
Let us use for the network hardware implementation
Altera FPGA Cyclone III EP3C16Q240 integral circuit.
The circuit consists of the following main parts: the
Configurable Logic Block, which provides physical
support for the program downloaded into the FPGA;
Input-Output
Block
possesses
communication
possibilities outside the FPGA; Programmable
Interconnection Block connects the different parts of
FPGA and allows them to communicate with each other
(Fig. 4). Since the indicated connections are
programmable therefore combining them is a part of
implementation process.
“COMPUTER SCIENCE & INFORMATION TECHNOLOGIES” (CSIT’2014), 18-22 NOVEMBER 2014, LVIV, UKRAINE
4
Fig.4 FPGA hardware implementation architecture of
the network.
The hardware design starts from developing a bridge
between host-PC and FPGA-board via parallel interface.
Training data are stored in shift registers and passes
XNOR logic gates, where they are mixed with constant
values of Classes C in lpm_constant blocks. All external
threads are controlled by the ocp_temeout block that
also has a function of clocking internal logic and
arithmetics elements. The main feature of the architecture
presented in Fig. 4 is conversion of internal data flows
from parallel form into serial stream. As far as the FPGA
is capable ещ perform distributed tasks, conversion
operates in a real time and does not affect the overall
system performance. The described architecture also
fulfills a function of muxing and demuxing the processed
signals and distributing them to different levels of the
system. Hard-level design of the network is implemented
by using hardware-defined graphical language. Each
functional block of the WTA neural circuits is realized by
using Verilog language stored in HDL-blocks. In such
mode a scalability and real-time signal processing are
provided.
Conclusion
The designed network implemented in hardware
comparatively to its software implemented counterparts
will offer such advantages as speed, cost, graceful
degradation, and compactness [13], [14].
The error of classification of static input features by the
SONN for some test Wine data is larger than 6% against
its less than 1.5% value for FSM method. Moreover, the
error of classification by SONN for some test Iris data is
higher than 5% in contrast to 4% value obtained by using
the method of k-nearest neighbors [3]. Therefore the
problem of precision rising of classification by SONNs
has to be solved. This problem is supposed to solve by
using SONN with higher order nonlinearities
comparatively to existing ones. Such the problem is a
subject of further investigations of the authors.
References
R. (eds.): Biocybernetics and Biomedical Engineering.
EXIT, Warszawa (2000).
[2] Adrian Horzyk, Self Optimizing Neural Networks
SONN-3 for Classification Tasks, Proc. of HAIS
2008, LNAI 5271, eds. E. Corchado, A. Abraham, W.
Pedrycz, Springer-Verlag, Berlin Heidelberg 2008,
pp. 229-236.
[3] Adrian Horzyk, Introduction to Constructive and
Optimization Aspects of SONN-3, Proc. of ICANN
2008, LNCS 5164, eds. V. Kurkova et al., SpringerVerlag, Berlin Heidelberg 2008, pp. 763-772.
[4] S. M. Lee and S. J. Roberts, “Sequential Dynamic
Classification Using Latent Variable Models”,
Technical Report PARG 08-02, Robotics Research
Group of Engineering Science Department of Oxford
University, 2008.
[5] J. W. Yoon,_S. J. Roberts, M. Dyson, J. Q. Ganb,
“Adaptive classification for Brain Computer Interface
systems using Sequential Monte Carlo sampling,”
Neural Networks, vol. 22, pp. 1286-1294, June 2009.
[6] R. P. Lippmann, “An introduction to computing with
neural nets,” IEEE Acoustics, Speech and Signal
Processing Magazine, vol. 3, no. 4, pp. 4-22, Apr.
1987.
[7] E. Majani, R. Erlanson, and Y. Abu-Mostafa, “On the
k-winners-take-all network,” In Advances in Neural
Information Processing Systems, vol. 1, D. S.
Touretzky, Ed. San Mateo, CA: Morgan Kaufmann,
1989, pp. 634–642.
[8] P. V. Tymoshchuk, “A simplified continuous-time
model of analogue K-winners-take-all neural circuit”,
in Proc. XI Int. Conf. “The Experience of Designing
and Application of CAD Systems in Microelectronics
Polyana-Svalyava, Ukraine, February 23-25, 2011,
pp. 121-125.
[9] A. Cichocki and R. Unbehauen, Neural Networks for
Optimization and Signal Processing (New York: John
Wiley and Sons, 1993).
[10] Suhap Sahin, Yasar Becerikli, and Suleyman Yazici,
Neural Network Implementation in Hardware Using
FPGA, Proc. of ICONIP 2006, LNCS 4234, eds. I.
King et al., Springer-Verlag, Berlin Heidelberg
2006, pp. 1105-1112.
[11] A. Muthuramalingam, S. Himavathi and E.
Srinivasan, “Neural network implementation using
FPGA: issues and application”, International
Journal of Information Technology, vol. 4, no 2,
2008, pp. 95-101.
[12] M. Krips, T. Lammert and A. Kummert, “FPGA
implementation of a neural network for a real-time
hand tracking system”, Proceedings of the 1st IEEE
International Workshop on Electronic Design, Test
and Applications, vol. 29-31, 2002, pp. 313-317.
[13] M. Hanggi, G. Moschytz, Cellular Neural Networks:
Analysis, Design, and Optimization, Kluwer
Academic Publishers, Norwell, MA, USA, 2000.
[14] J. Misra and I. Saha, ”Artificial neural networks in
hardware: A survey of two decades of progress”,
Neurocomputing, vol. 74, 2010, pp. 239-255.
[1] Duch, W., Korbicz, J., Rutkowski, L., Tadeusiewicz,
“COMPUTER SCIENCE & INFORMATION TECHNOLOGIES” (CSIT’2014), 18-22 NOVEMBER 2014, LVIV, UKRAINE