A Self-Organizing Fuzzy Neural
Networks
H. S. LIN, X. Z. GAO, XIANLIN HUANG,
AND Z. Y. SONG
Abstract
† This paper proposes a novel clustering
algorithm for the structure learning of fuzzy
neural networks. Our clustering algorithm
uses the reward and penalty mechanism for
the adaptation of the fuzzy neural networks
prototypes at every training sample.
Compared with the classical clustering
algorithms, the new algorithm can on-line
partition the input data, pointwise update the
clusters, and self-organize the fuzzy neural
structure.
Abstract
† All rules are self-created, and they grow
automatically with more incoming data. There
are no conflicting rules in the created fuzzy
neural networks. Our approach shows
supervised clustering algorithms can be
suitable for the structure learning of the selforganizing fuzzy neural networks. The
identification of several typical nonlinear
dynamical systems and prediction of time
series data are employed to demonstrate the
effectiveness of the proposed fuzzy neural
networks and its learning algorithm.
Introduction
† It is well known fuzzy logic provides human
reasoning capabilities to capture uncertainties,
which cannot be described by precise
mathematical models. Neural networks offer
remarkable advantages, such as adaptive
learning, parallelism, fault tolerance, and
generalization. They have been proved to be
powerful techniques in the discipline of
system control, especially when the
controlled system is difficult to be modeled
accurately, or the controlled system has large
uncertainties and strong nonlinearities.
Introduction
† Therefore, fuzzy logic and neural networks
have been widely adopted in model-free
adaptive control of nonlinear systems.
† There are numerous kinds of neural fuzzy
systems proposed in the literature，and most
of them are suitable for only off-line cases.
Some on-line learning methods for the neural
fuzzy systems are studied as well.
Introduction
† In this paper, we proposed a novel on-line
clustering algorithm of structure learning for
our fuzzy neural networks. This new
clustering algorithm employs the mechanism
of reward and penalty action used in the
Learning Vector Quantization (LVQ). Our
fuzzy neural networks with the on-line
structure and parameter learning capability is
a suitable candidate for real-time applications
due to its fast convergence.
Structure of Fuzzy Neural
Networks
y1 y2 d
y1d
x1
y2
x2
Structure of Fuzzy Neural
Networks
† Layer 1: Each node in this layer, only
transmits input values to the next layer
directly. Thus, the function of the th node is
defined as
f = ui1 = xi
a= f
† Layer 2: Each node in this layer corresponds
to one linguistic label of one of the input
variables in Layer 1. The operation performed
in this layer is
(2)
1 ui − mij 2
)
f =− (
σ ij
2
a = ef
Structure of Fuzzy Neural
Networks
† Layer 3: Nodes in this layer are rule nodes,
and constitute the antecedents of the fuzzy
rule base. The input and output functions of
the th rule node
are
n
f = ∏ ui(3)
i =1
a= f
† Layer 4: The nodes in this layer are called
“output-term nodes”. The links in Layer 4
perform the fuzzy OR operation that have the
same consequent
f =
J
∑
j =1
u 4j
a = min(1, f )
Structure of Fuzzy Neural
Networks
† Layer 5: These nodes and Layer 5 links
attached act as the defuzzifier. The following
functions can perform the Center Of Area
(COA) defuzzification method:
f = ∑ w u = ∑ (mijσ ij )u
a =
f
∑σ u
† Based on the above structure, an on-line
learning algorithm will be proposed to
determine the proper centers and widths of
term nodes.
5 5
ij i
5
i
ij
5
i
Learning Algorithm for
Structure Identification
† In our fuzzy neural networks, for every online incoming training pattern, we first use
the novel cluster algorithm to identify the
structure,
and
next
apply
the
backpropagation algorithm to optimize the
parameters. In our learning method, only the
training data is need. The input /output-term
nodes and rule nodes are created dynamically
as learning proceeds upon receiving on-line
incoming training data. During the learning
process, novel input-term and output-term
nodes and rule nodes will be added.
Learning Algorithm for
Structure Identification
† The main idea of our clustering algorithm is for
every input data, we first find the winner
clusters in the input and output space
respectively. Next, as in the fuzzy ARTMAP, we
check that if the winner cluster in the input
space is connected to the winner cluster in the
output space. If so, we assume that the winner
cluster in the output space is the correct
prediction of the winner cluster in the input
space, which is analogous to the fact the fuzzy
ARTb category activated by the input is the
correct prediction of the fuzzy ARTa categories
activated by an input in the fuzzy ARTMAP.
Learning Algorithm for
Structure Identification
† If not, we assume that the mismatch occurs
between the winner cluster in the input space
and the winner cluster in the output space
and we will begin to search for another
cluster in the input space which will match
the winner cluster in the output space . The
reward and penalty mechanism is employed
in our clustering algorithm. We can describe
the novel clustering algorithm as follows.
Learning Algorithm for
Structure Identification
† Step1: Initialize the fuzzy system with zero
cluster: In = 0 , On = 0 .
† Step 2: For the first input and output training
vectors, they are selected as the centers of
the first clusters in the input and output
space, respectively. We connect the first
cluster in the input space to the first cluster
in the output space, and set the number of
data belonging to the cluster as one.
Learning Algorithm for
Structure Identification
† Step 3: For an input data point in the input
space, we compute the distances between
the input vector and the existing input space
clusters using the Euclidean metric function:
q
d p = ∑ xil − I _ mlp
2
i =1
The nearest cluster (winner neuron) is
chosen by selecting the minimum d . Next,
the following algorithms are used:
j
Learning Algorithm for
Structure Identification
† If d j is larger than a certain value d vigilance , we
assume that this input data does not belong
to any existing cluster, we form a novel
cluster. The newly added cluster is the winner
cluster in the input space. If d j is smaller
than d
, we assume that cluster j is the
winner cluster in the input space. The
procedure in the input space in Step 3 is also
adopted in the output space. We can also find
the winner cluster in the output space.
vigilance
Learning Algorithm for
Structure Identification
† Step 4: We check the mapping process from
input clusters to the output clusters.
(1). If the winner cluster in the input space is a
novel cluster, we connect this novel cluster to
the winner cluster in the output space, and
update the parameters of the winner cluster
in the output space.
eph _ Omwinner =
Omwinner × Ocwinner + x
Ocwinner + 1
Learning Algorithm for
Structure Identification
Oσ
2
winner
2
2
Ocwinner × (Oσ winner
+ Om winner
) + y2
2
=
− eph _ Om winner
Ocwinner + 1
Omwinner = eph _ Omwinner
Ocwinner = Ocwinner + 1
† (2). If the winner cluster of the input space is
connected to the winner cluster of the output
space originally, we adopt the following
algorithm to update the centers (), variances
and the counters of the winner cluster in the
input space.
Learning Algorithm for
Structure Identification
eph _ Imwinner =
Imwinner × Icwinner + x
Icwinner + 1
Imwinner = eph _ Imwinner
Iσ
2
winner
2
2
Icwinner × ( I σ winner
+ Imwinner
) + x2
2
=
− eph _ Imwinner
Icwinner + 1
Icwinner = Icwinner + 1
† (3). If the winner cluster of the input space is
not connected to the winner cluster of the
output space yet, we use the following
algorithm to punish the winner cluster of the
input space
Learning Algorithm for
Structure Identification
eph _ Imwinner =
Imwinner × Icwinner − x
Icwinner − 1
Imwinner = eph _ Imwinner
2
I σ winner
=
2
2
Icwinner × ( I σ winner
+ Imwinner
) − x2
2
− eph _ Imwinner
Icwinner − 1
Icwinner = Icwinner
† After that, we return to Step 3 to search for
another cluster in the input space which will
match the winner cluster in the output space
Learning Algorithm for
Structure Identification
† Our structure learning algorithm is actually a
supervised clustering method for identifying the
structure of the fuzzy neural networks. As we know,
supervised cluster algorithms are indeed effective,
and they converge fast. Furthermore, our fuzzy neural
networks has the remarkable self-learning ability.
That is, it can self-generate fuzzy rules and self-adapt
the structure and synaptic weights. Note, there are no
conflicting rules in the generated fuzzy neural
networks. In summary, the proposed structure
learning strategy provides a new way to utilize a class
of supervised clustering algorithms for the on-line
structure learning of fuzzy neural networks.
Learning Algorithm for
Structure Identification
† Our structure learning algorithm is actually a
supervised clustering method for identifying the
structure of the fuzzy neural networks. As we know,
supervised cluster algorithms are indeed effective,
and they converge fast. Furthermore, our fuzzy neural
networks has the remarkable self-learning ability.
That is, it can self-generate fuzzy rules and self-adapt
the structure and synaptic weights. Note, there are no
conflicting rules in the generated fuzzy neural
networks. In summary, the proposed structure
learning strategy provides a new way to utilize a class
of supervised clustering algorithms for the on-line
structure learning of fuzzy neural networks.
Parameter Learning of Fuzzy
Neural Networks
† We use the backpropagation algorithm to
tune the parameters of the fuzzy neural
networks.
The centers and variances of the cluster in
Layer 5 are updated by
Oσ k (t ) × f k4 (t )
Omk (t + 1) = Omk (t ) + η × [ y1(t ) − y (t )] ×
f 1(t )
4
4
Oσ k (t + 1) = Oσ k (t ) + η × [ y1(t ) − y (t )] × Omk (t ) × f k (t ) × f 21(t ) − f 2(t ) × f k (t )
f 1 (t )
Parameter Learning of Fuzzy
Neural Networks
† The centers and variances of the cluster in
Layer 2 are updated by
Imij (t + 1) = Imij (t ) + η × errorj3 (t ) × f j3 (t ) × 2 ×
I σ (t + 1) = I σ (t ) + η × error (t ) × f (t ) × 2 ×
i
j
i
j
3
j
3
j
xi − Imij (t )
( Iσ ij (t )) 2
( xi − Imij (t )) 2
( I σ ij (t ))3
Simulations
† Example 1 — Identification of an SISO
dynamic system: The plant to be identified is
described by the following difference equation
y (t + 1) =
y (t )
+ u 3 (t ).
2
1 + y (t )
Simulations
0.25
RMS errors
0.2
0.1
0
0
20000
40000
60000
Iterations
80000
100000
† Root-mean-square errors during learning.
Simulations
1.5
1
Output
0.5
0
-0.5
-1
-1.5
0
50
100
150
200
250
Time in Samples
300
350
400
† Outputs of SISO system and identification model.
Simulations
† Example 2 — Identification of an MISO
dynamic system: the plant to be identified is
a two-input and one-output dynamic system
described by the following equations:
y = ( x12 − x2 ) 2 + (1 − x1 ) 2
Simulations
6
5
Outputs
4
3
2
1
0
0
10
20
30
40
50
60
Time in Samples
70
80
90
100
† Outputs of MISO system and identification
Simulations
† Example 3 — Identification of the MIMO
dynamic system: the plant is described as:
⎡ y1 (t ) ⎤
2
⎢
⎥
⎡ y1 (t + 1) ⎤ ⎢1 + y2 (t ) ⎥ ⎡u1 (t ) ⎤
⎢ y (t + 1) ⎥ = ⎢ y (t ) y (t ) ⎥ + ⎢u (t ) ⎥
1
2
⎣ 2
⎦
⎣ 2 ⎦
⎢
⎥
2
⎣ 1 + y2 (t ) ⎦
Simulations
4
3
2
1
y1
0
-1
-2
-3
-4
-5
-6
†
0
10
20
30
40
50
60
Time in Samples
70
80
90
100
y1 response of MIMO system and identification model.
Simulations
3
2
y2
1
0
-1
-2
-3
†
0
10
y 2 response
20
30
40
50
60
Time in Samples
70
80
90
100
of MIMO system and identification model.
Simulations
† Example 4 — Prediction of time series.
The performance of our fuzzy neural networks
in dealing with real-world prediction problems
is demonstrated here by predicting the time
series of an automobile gearbox. This slightly
nonlinear time series is collected using a
sound level meter, and the analyzed acoustic
data is provided by a Japanese automobile
manufacturer.
Simulations
† It represents the external sound level of an
automatic transmission system. Below is a brief
summary of the experimental measurement
conditions and used instrumentation:
①．1000 r/min rotational velocity of the
automobile engine,
②．5 Nm load torque,
③．Integrated sound level meter (Ono Sokki LA5110),
④．5 ms sampling period.
Simulations
3
2
y2
1
0
-1
-2
-3
0
10
20
30
40
50
60
Time in Samples
70
80
90
100
† Actual and predicted time series of gearbox.
Conclusions
† In our paper, a new novel clustering
algorithm is proposed for the structure
learning of the fuzzy neural networks. This
clustering algorithm can on-line partition the
input data and self-organize the fuzzy neural
structure. Therefore, no priori knowledge of
the distribution of the input data is needed
for initialization of fuzzy rules. They are
automatically generated with the incoming
training data.
Conclusions
† Our fuzzy neural networks can use this online training algorithm for the structure and
parameter training. The effectiveness of the
proposed learning algorithm is verified by the
identification of dynamical nonlinear systems
and prediction of time series.