Applications of A Fuzzy-Logic-Controlled Neural Network:
Optimization of Training and Performance
Shu-Fai Au and Patricia A. Nava, Ph.D.
Department of Electrical and Computer Engineering
University of Texas at El Paso
El Paso, TX 79968
Voice: (915) 747-5994
Fax: (915) 747-7871
Email: fau@ece.utep.edu
Email: pnava@ece.utep.edu
ABSTRACT: One major problem in the use of
neural networks is the long training time. The purpose of this paper is to demonstrate the optimization of training that occurs with the application of
fuzzy logic controller theory to neural networks.
The resulting fuzzy-logic-controlled neural network (FLCNN) exhibits a significant cut in the
training period. A fuzzy logic system (FLS) is employed to control the learning parameters of a neural network (NN) to reduce the possibility of overshooting during the learning process. Hence, the
learning time of the neural network can be shortened. This paper compares the training efficiency
and accuracy between a NN and a FLCNN, when
they are required to carry out the same assignment.
In one application, the training time is reduced by
30%.
1. INTRODUCTION
Neural networks have been proven to be very
efficient at handling a wide range of engineering
applications [1]. The use of neural networks is
enticing because a solution is obtained by allowing
the network to “learn” instead of designing a solution. A formal training procedure is required by the
neural network in order to learn.
One supervised training procedure, the backpropagation algorithm, has become one of the most
widely used procedures due to its simple mathematical operation. However, the back-propagation
algorithm has a major drawback: a long period of
training time. That is, for any given problem, the
network must have many solved examples presented repeatedly during the learning procedure before
the network learns the data with an acceptable degree of accuracy. Therefore, the learning stage is a
lengthy process. This process can be shortened by
the use of fuzzy logic controllers.
Fuzzy logic controllers have structured, numerical knowledge that can exploit noisy or inexact situations [2]. By employing a fuzzy logic control system to adaptively vary the learning parame-
ters of the network, the training time can be reduced significantly. The fuzzy logic control system
uses “expert knowledge” to determine the specific
change in learning parameters. The following sections describe, in detail, how the learning process
is streamlined. In addition to the training efficiency, the neural network still obtains an impressive
performance, which is evidenced by the experimental results, discussed in section 4.2.
2. NEURAL NETWORKS
Multilayer feedforward neural networks have
proven their abilities in solving and handling a
wide range of problems and applications, and these
systems have overcome limitations of the single
layer system. In order to “learn” a solution, a training strategy must be used. One of the most popular
training methods is backpropagation.
2.1 Backpropagation Training
In a typical backpropagation training session
for a multilayer network, the supervisor presents
input data to the network and compares the network’s actual output, Op, to the target (or desired)
output, Tp:
E = (Tp – Op)2/2
(1)
This difference, or error, is used to change the
connection weights between neurons in the network:
wj(t + 1) = (-∂E / ∂wj) + wj(t)
(2)
In this manner, the network learns from the data.
Once trained, the network interconnection weights
are static, and testing (new, unseen) data is presented to the network. With the backpropagation
algorithm, neural network weights are adjusted in a
gradient descent manner, as illustrated in Figure 1,
RMS error
RMS error =
Initial error

p
k
(t kp  x kp )
2
n p no
(3)
Finally, the NN may complete its learning, and
possibly take thousands iterations of data presentations.
X
Initial error
Z
Global minimum
Figure 1. A neural network converging to
a global minimum.
minimum
Global minimum
which means the minimization of the error between the expected output and the actual output for
a particular input. However, two major problems
exist with this kind of network. First, slow convergence rate and second, the possibility of settling
into a local minimum. A potentially large number
of iterations are required to train the network until
it learns the data with an acceptable degree of accuracy. A method of shortening the training time is
desirable and exists in the form of a fuzzy logic
controller in the role of supervisor.
2.2 Standard Training Shortcomings
One major problem with a back-propagation
neural network is its slow convergence time [3].
Let’s picture the following scenario in a 3-D space.
During training, a NN is trying to converge to the
global minimum with the shortest path. Global
minimum is the maximum performance that the
neural network can yield given the size of the network. At the beginning of a learning process, the
network learns rapidly, and its value of RMS error
decreases fast. In general, the value of RMS reflects how a neural network is performing, and
computed by equation (3). The learning process
starts to slow down after some point in time, as the
NN starts overshooting the global minimum, as
illustrated in Figure 2.
Figure2: A neural network overshoots a global
minimum.
3. FUZZY LOGIC CONTROLLERS
A fuzzy logic control system consists of four
components as shown in Figure 3 [4]. A fuzzifier
converts crisp inputs into fuzzy numbers and associates them with membership value . The fuzzy
rule base contains the relations between the input
and output, and the applications of these rules are
called firing. The inference engine combines the
output of all rules. And, the defuzzifier converts
the fuzzy numbers back to a crisp value.
There are two reasons that fuzzy logic control
systems are preferred: the infinite measurement
resolution, and imprecise linguistic descriptions.
Because of the use of fuzzy sets and linguistic variables, there is no limited resolution as it does in a
conventional control system. “Forced efficiency”
on the output can be the result caused by low resolution. Linguistic variables are variables whose
values are expressed in words or sentences. The
concept was introduced by Zadeh [5] to provide a
method to characterize some situations that are too
complex to be defined with conventional
Input
Fuzzy logic
controller
Fuzzifier
Fuzzy rule
base
Inference
engine
Defuzzifier
Output
Figure 3. A general scheme of a Fuzzy
Logic Controller
quantitative terms. An example of the use of linguistic variables is shown in Figure 4 [6]. “Slow,”
“moderate” and “fast” in relation to speed, in miles
per hour, are shown with their respective categories and membership values. In addition to using
linguistic variables, fuzzy logic control systems
can encapsulate complex input/output operations
in the form of rules.
Also, the operations of most control systems
are based on “expert knowledge”. “Expert
knowledge” is the experience gained by a human
operator about the input and output relationships of
the system, and it might be very difficult to express
in precise terms.
4. FUZZY LOGIC CONTROLLED NEURAL
NETWORK
The implementation of fuzzy logic control
system to adaptively determine learning parameters in neural networks is discussed in this section.
The experimental results are presented, too.
4.1 Implementation
A fuzzy logic control system to adaptively
determine learning parameters (i.e. learning rate
and momentum) in neural networks is the focus of
this study. As proposed by Choi [7], this contrasts
with regular neural networks, in which the learning
parameters are fixed. The on-line fuzzy logic controller is utilized to adapt the learning parameters
based on the RMS error generated by the neural
network. A basic architecture for such an integrated system is shown in Figure 5.
The purpose of the FLC is to automatically
adjust the learning rate  and momentum term
according to the RMS error surface. To describe
the error surface, there are four parameters used to
create the rules for the FLC. First, the change in
error (CE) described as
CE(t) = error(t) – error(t-1),
(5)
second, the change in CE (CCE):
CCE = CE(t) – CE(t-1),
(7)
third, the sign change in error (SC):
SC(t) = 1 - || ½[sgn(CE(t-1) + sgn(CE(t))] ||,
(8)
And lastly, the cumulative sum of SC (CSC):
Slow
Moderate
Fast
t
CSC(t) =
 SC (m).
(9)
m t  4

are used.
The collection of rules created (called the
rule base) reflects the use of these parameters to
adaptively change the neural network’s learning
rate and momentum. To demonstrate the idea, two
of the fuzzy IF-THEN rules that are employed in
the FLC are listed:
0
10 20 30 40 50 60 70 80
mph
Figure 4: An example of linguistic variables.
RULE 1:
IF CE is small AND CSC is less or equal to
two, THEN the value of learning parameters
should be increased.
RULE 2:
IF CSC is larger than three, THEN the value of
learning parameters should be decreased regardless
the value of CE and CCE.
Target Output
Training
samples
Actual
Output
Table 2: Decision table for ∆, when CSC  2.
CE
CCE
NL
NS
ZE
PS
PL
NL
NS
ZE
PS
PL
-0.01
-0.01
0
0
0
-0.01
0
0.01
0
0
0
0
0.01
0
0
0
0
0.01
0
-0.01
0
0
0
-0.01
-0.01
Neural
Network
Error
Learning
Parameters
Fuzzy logic
Controller
Figure 5: A fuzzy logic controlled neural network.
The rule base can be summarized concisely in the
form of a decision table. Two of the fuzzy rule
bases (decision tables) for the applications discussed in section 4.2 are shown below. Table 1
describes the rule base for the change in the learning parameter, ∆, and Table 2 describes the rule
base for the change in momentum, ∆. In these
decision tables, the utilization of five linguistic
variables has been used. Specifically, NL represents a “Negative Large” value, NS is “Negative
Small,” ZE is zero, PS is “Positive Small,” and PL
is “Positive Large.”
4.2 Application to Letter Recognition
The system was tested on a classic
application area, letter recognition. The FLCNN
showed a 30% average improvement in training
time [8]. That is, compared the performance of a
regular neural network with different value
learning rate, and momentum term trained in the
normal fashion, the fuzzy logic controlled system
requires about 2/3 of the iterations (measured to
convergence by an error limit). Futhermore, the
fuzzy
logic
controlled
system
assesses
performance with each iteration, and the
performance is actually improved. In one specific
case when both systems have a learning rate of 0.7
and momentum term of 0.7, the regular NN fails to
learn in about 100 iterations as shown in next two
figures, while the FLCNN system is able to
continue learning.
Rms
error
0.8
0.7
0.6
0.5
0.4
Table 1: Decision table for ∆, when CSC  2.
CE
CCE
NL
NS
ZE
PS
PL
NN
0.3
0.2
NL
NS
ZE
PS
PL
NS
NS
ZE
NS
NS
NS
ZE
PS
ZE
NS
NS
PS
ZE
PS
NS
NS
ZE
NS
ZE
NS
NS
NS
ZE
NS
NS
0.1
FLCNN
0
0
50
100
150
200
250
iterations
Figure 7: RMS error of FLCNN vs. NN with  =
0.7 and  = 0.7.
FLCNN
Accuracy,
%
90
80
70
60
50
40
30
NN
20
10
0
0
50
100
150
200
250
iterations
Figure 8: Accuracy of FLCNN vs. NN with  =
0.7 and  = 0.7.
5. Conclusions
This study demonstrated that the neural network’s training time is reduced dramatically by
employing a fuzzy logic controller to adaptively
vary the learning parameters. When this method is
applied for the letter recognition task, it yields a
92% accuracy, which is a better performance than
the originally proposed approach, a Holland-style
classifier. In addition, this technique can reduce
the possibility of overshooting and sometimes help
the network get out of a local minimum [8]. The
network’s ability to converge during training and
the final performance are dependent on the learning parameters. Our study reinforces this fact, as
our simulations have demonstrated that a “wrong”
value of learning rate can lead to poor letter recognition accuracy. Furthermore, the methodology is
portable, and can be exercised on other neural
network applications.
6. References
[1]Zurada, J., An Introduction to Artificial Neural
Systems, West Publishing Company, St. Paul,
Minnesota, 1992.
[2] Kosko, B., “Fuzzy systems as universal approximators,” Proc. Of the IEEE Int. Conference
on Fuzzy Systems, San Diego, CA, p. 1153-1162,
1992.
[3]Dayhoff, J.E., Neural Network Architecture: An
Introduction, Van Nostrand Reinhold, New York,
NY, 1990.
[4] Kosko, Bart, Neural Networks and Fuzzy Systems: A Dynamical Approach to Machine Intelligence, Prentice Hall, Englewood Cliffs, NJ, 1992.
[5] Zadeh, L.A., Fuzzy Sets and Applications: Selected Papers by L. A. Zadeh, John Wiley & Sons,
New York, 1987.
[6] Klir, G.J., and Yuan, B., Fuzzy Sets and Fuzzy
Logic: Theory and Applications, Prentice Hall Inc.,
Upper Saddle River, NJ, 1995.
[7] Lin, C.T., and Lee, C.S., Neural Fuzzy Systems: A Neuro-Fuzzy Synergism to Intelligent Systems, Prentice Hall Inc., Upper Saddle River, NJ,
1996.
[8] Au, S.F., “Letter Recognition using a Fuzzy
Logic Controlled Neural Network,” Master’s Thesis, University of Texas at El Paso, 1998.