Genetic Design of GMDH-type Neural Networks for Modelling of
Thermodynamically Pareto Optimized Turbojet Engines
a
a
K. Atashkari, aN. Nariman-zadeh , aA. Darvizeh, bX. Yao, aA. Jamali, aA. Pilechi
Department of Mechanical Engineering,Engineering Faculty, The University of Guilan
P.O. Box 3756, Rasht, IRAN
b
School of Computer Science, The University of Birmingham
Edgbaston, Birmingham B15 2TT, U.K.
Abstract:- Firstly, a multi-objective genetic algorithm (GAs) is used for Pareto based optimization of
thermodynamic cycle of ideal turbojet engines considering four important conflicting thermodynamic objectives,
namely, specific thrust (ST), specific fuel consumption (SFC), propulsive efficiency (p), and thermal efficiency
(t). This provides the best Pareto front of such four-objective optimization from the space of design variables,
which are Mach number and pressure ratio, to the space of the above-mentioned four thermo-mechanical
objective functions. Secondly, genetic algorithms (GA) with a new encoding scheme are used for optimal design
of both connectivity configuration and the values of coefficients, respectively, involved in GMDH-type neural
networks for the inverse modelling of the input-output data table obtained as the best Pareto front.
Key-Words:- Turbojet Engines, Pareto, Group Method of Data Handling (GMDH), Genetic Algorithms (GAs).
1
Introduction
In the optimization of complex real-world problems,
there are several objective functions or cost functions
(a vector of objectives) to be optimized (minimized
or maximized) simultaneously. These objectives
often conflict each other so that improving one of
them will deteriorate another objective function.
Therefore, there is no single optimal solution as the
best with the respect to all the objective functions.
Instead, there is a set of optimal solutions, well
known as Pareto optimal solutions or Pareto front
[1], which distinguishes significantly the inherent
natures between single-objective and multi-objective
optimization problems. The concept of Pareto front
or set of optimal solutions in the space of objective
functions in multi-objective optimization problems
(MOPs) stands for a set of solutions that are nondominated to each other but are superior to the rest
of solutions in search space. It must be noted that
this non-dominancy does exist in different levels,
forming different ranked Pareto fronts, although the
first Pareto front is the most important and will be
the ultimate solution. The inherent parallelism in
evolutionary algorithms makes them suitably eligible
for solving MOPs. Among these methods, the Vector
Evaluated Genetic Algorithm (VEGA) proposed by
Schaffer [2], Fonseca and Fleming’s Genetic
Algorithm (FFGA) [2], Non-dominated Sorting
Genetic Algorithm (NSGA) by Srinivas and Deb [1],
and Strength Pareto Evolutionary Algorithm (SPEA)
by Zitzler and Thiele [2], and the Pareto archived
evolution strategy (PEAS) by Knowles and Corne
[2] are the most important ones. A very good and
comprehensive survey of these methods has been
presented in [2]. Basically, both NSGA and FFGA as
Pareto-based approaches use the revolutionary nondominated sorting procedure originally proposed by
Goldberg [3]. Besides, the diversity issue and the
lack of elitism was also a motivation for
modification of that algorithm to NSGA-II [4] in
which a direct elitist mechanism, instead of sharing
mechanism, has been introduced to enhance the
population diversity. This modified algorithm has
been known as the state-of-the-art in evolutionary
MOPs.
System identification techniques are applied in many
fields in order to model and predict the behaviors of
unknown and/or very complex systems based on
given input-output data [5]. The Group Method of
Data Handling (GMDH) algorithm is a selforganizing approach by which gradually complicated
models are generated based on the evaluation of their
performances on a set of multi-input-single-output
data pairs (i=1, 2, …, M). The GMDH was firstly
developed by Ivakhnenko [6] as a multivariate
analysis method for complex systems modelling and
identification. The main idea of GMDH is to build an
analytical function in a feedforward network based
on a quadratic node transfer function [7] whose
coefficients are obtained using regression technique.
In fact, real GMDH algorithm in which model
coefficients are estimated by means of the least
squares method has been classified into complete
induction and incomplete induction, which represent
the combinatorial (COMBI) and multilayered
iterative algorithms (MIA), respectively [7]. In
recent years, the use of such self-organizing network
leads to successful application of the GMDH-type
algorithm in a broad range area in engineering,
science, and economics [7, 8].
DEFINITION OF PARETO DOMINANCE
A vector U  u1 , u2 ,..., uk   k is dominance to
vector V  v1 , v2 ,..., vk   k (denoted by U  V )
i 1,2,..., k , ui  vi 
j  1,2,..., k : u j  v j . In other words, there is at
if
In this paper, firstly, an optimal set of design
variables in turbojet engines, namely, the input Mach
number Mo and the pressure ratio of compressor c
are found using Pareto approach to multi-objective
optimization. In this study, four conflicting
objectives in ideal subsonic turbojet engines are
selected for optimization. These include thermal
efficiency (t) and propulsive efficiency (p)
together with specific fuel consumption (SFC) and
specific thrust (ST). Secondly, GAs are deployed in a
new approach to design the whole architecture of the
GMDH-type neural networks, i.e., the number of
neurons in each hidden layer and their connectivity
configuration, for modelling of the four-input-twooutput data table obtained in the first part of the
study. The connectivity configuration is not limited
to the adjacent layers, as have already proposed in
[8], so that general structure GMDH-type neural
networks, in which neurons in any layers can be
connected to each other, are evolved.
Multi-objective optimization which is also called
multicriteria optimization or vector optimization has
been defined as finding a vector of decision variables
satisfying constraints to give acceptable values to all
objective functions [2, 4]. In general, it can be
mathematically defined as:

only
if
least one u j which is smaller than v j whilst the rest
u ’s are either smaller or equal to corresponding v ’s.
DEFINITION OF PARETO OPTIMALITY
A point X *   (  is a feasible region in  n
satisfying equations (2) and (3)) is said to be Pareto
optimal (minimal) with respect to the all X   if
and only if F ( X * )  F ( X ) . Alternatively, it can be
readily restated as
i 1,2,..., k , X    { X *} f i ( X * )  f i ( X )
 j  1,2,..., k :
f j ( X * )  f j ( X ) . In other
words, the solution X * is said to be Pareto optimal
(minimal) if no other solution can be found to
dominate X * using the definition of Pareto
dominance.
DEFINITION OF PARETO SET
2 Multi-objective Optimization
*
find the vector X *  x1* , x*
2 ,..., xn
and
 to optimize
T
For a given MOP, a Pareto set Ƥ ‫٭‬is a set in the
decision variable space consisting of all the Pareto
optimal
vectors


Ƥ‫ ٭‬ { X   | ∄ X   : F ( X )  F ( X )} . In other
words, there is no other X  as a vector of decision
variables in  that dominates any X ∈Ƥ‫٭‬.
F ( X )  f1( X ), f 2 ( X ),..., f k ( X ) T ,
subject to m inequality constraints
(1)
DEFINITION OF PARETO FRONT
For a given MOP, the Pareto front ƤŦ fo tes a si ‫٭‬
vector of objective functions which are obtained
using the vectors of decision variables in the Pareto
gi ( X )  0 ,
(2)
set
Ƥ si
taht
,‫٭‬
ƤŦ‫ ٭‬ {F ( X )  ( f1 ( X ), f 2 ( X ),...., f k ( X )) : X  Ƥ‫٭‬

i  1 to m ,
and p equality constraints
h j ( X )  0 , j  1 to p ,

}. In other words, the Pareto front ƤŦ eht fo tes a si ‫٭‬
(3)
where X   is the vector of decision or design
variables, and F ( X )  k is the vector of objective
functions which each of them be either minimized or
maximized. However, without loss of generality, it is
assumed that all objective functions are to be
minimized. Such multi-objective minimization based
on Pareto approach can be conducted using some
definitions:
*
vectors of objective functions mapped from Ƥ.‫٭‬
n
3
Modelling Using GMDH-Type
Neural Networks
By means of GMDH algorithm a model can be
represented as set of neurons in which different pairs
of them in each layer are connected through a
quadratic polynomial and thus produce new neurons
in the next layer. Such representation can be used in
modelling to map inputs to outputs. The formal
definition of the identification problem is to find a
function fˆ so that can be approximately used instead
of actual one, f , in order to predict output ŷ for a
it is now possible to train a GMDH-type neural
network to predict the output values ŷ for any given
In the basic form of the GMDH algorithm, all the
possibilities of two independent variables out of total
n input variables are taken in order to construct the
regression polynomial in the form of equation (8)
that best fits the dependent observations ( y i , i=1, 2,
…, M) in a least-squares sense. Consequently,
 n  n(n  1)
  
neurons will be built up in the first
2
 2
hidden layer of the feedforward network from the
observations { ( y i , x , x ); (i=1, 2, …, M)} for
input vector X  ( x , x , x ,..., x ) , that is
different p, q  {1,2,..., n} [7]. In other words, it is
given input vector X  ( x , x , x ,..., x ) as close as
1
2
3
n
possible to its actual output y . Therefore, given M
observation of multi-input-single-output data pairs so
that
(i=1,2,…,M), (4)
y  f ( x , x , x ,..., x )
i
i1 i 2
i3
in
i
i1
i2
yˆ  fˆ ( x , x , x ,..., x )
i
i1 i 2 i 3
in
i3
in
(i=1,2,…,M).
(5)
The problem is now to determine a GMDH-type
neural network so that the square of difference
between the actual output and the predicted one is
minimised, that is
M
2
 [ fˆ ( x , x , x ,..., x )  y ]  min
i1 i 2 i 3
in
i
i 1
.
(6)
General connection between inputs and output
variables can be expressed by a complicated discrete
form of the Volterra functional series in the form of
n
n n
n n n
y  a 0   ai xi    aij xi x j     aijk xi x j x k ...
i 1
i 1 j 1
i 1 j 1 k 1
(7)
which is known as the Kolmogorov-Gabor
polynomial [7]. This full form of mathematical
description can be represented by a system of partial
quadratic polynomials consisting of only two
variables (neurons) in the form of
yˆ  G ( xi , x j )  a 0  a1 xi  a 2 x j  a3 xi x j  a 4 xi
2
 a5 x j
2
.
(8)
In this way, such partial quadratic description is
recursively used in a network of connected neurons
to build the general mathematical relation of inputs
and output variables given in equation (7). The
coefficient a i in equation (8) are calculated using
regression techniques [7] so that the difference
between actual output, y, and the calculated one, ŷ ,
for each pair of x i , x j as input variables is
minimized. Indeed, it can be seen that a tree of
polynomials is constructed using the quadratic form
given in equation (8) whose coefficients are obtained
in a least-squares sense. In this way, the coefficients
of each quadratic function G are obtained to
i
optimally fit the output in the whole set of inputoutput data pair, that is
M
2
 ( y  G ())
i
i
E  i 1
 min
M
.
(9)
ip
iq
now possible to construct
M
data triples
{ ( yi , xip , xiq ); (i=1, 2, …, M)} from observation
using such p, q  {1,2,..., n} in the form
 x1 p

 x2 p
 x Mp

x1q
x 2q
x Mq
y1 

y2  .
y M 
Using the quadratic sub-expression in the form of
equation (8) for each row of M data triples, the
following matrix equation can be readily obtained as
(10)
Aa  Y
where a is the vector of unknown coefficients of
the quadratic polynomial in equation (8)
a  {a 0 , a1 , a 2 , a 3 , a 4 , a 5 }
(11)
and
(12)
Y  { y1 , y 2 , y 3 ,..., y M }T
is the vector of output’s value from observation. It
can be readily seen that
1

A  1
1

x1 p
x2 p
x Mp
x1q
x 2q
x Mq
x1 p x1q
x 2 p x 2q
x Mp x Mq
x12p
x 22 p
2
x Mp
x12q 

x 22q  .
2 
x Mq

(13)
The least-squares technique from multiple-regression
analysis leads to the solution of the normal equations
in the form of
(14)
a  ( AT A) 1 AT Y
which determines the vector of the best coefficients
of the quadratic equation (8) for the whole set of M
data triples. It should be noted that this procedure is
repeated for each neuron of the next hidden layer
according to the connectivity topology of the
network.
4
Application
of
Genetic
Algorithm in the Topology Design of
GMDH-type Neural Networks
Evolutionary methods such as genetic algorithms
have been widely used in different aspects of design
in neural networks because of their unique
capabilities of finding a global optimum in highly
multi-modal and/or non-differentiable search space
[8-9]. Such stochastic methods are commonly used
in the training of neural networks in terms of
associated weights or coefficients which have
successfully performed better than traditional
gradient-based techniques [9]. The literature shows
that a wide range of evolutionary design approaches
either for architectures or for connection weights
separately, in addition to efforts for them
simultaneously [9]. In the most GMDH-type neural
network, neurons in each layer are only connected to
neuron in its adjacent layer as it was the case in
Methods I and II previously reported in [8]. Taking
this advantage, it was possible to present a simple
encoding scheme for the genotype of each individual
in the population as already proposed by authors [8].
the second hidden layer and used with abbc in the
same layer to make the output neuron abbcadad as
shown in the figure (1). It should be noted that such
repetition occurs whenever a neuron passes some
adjacent hidden layers and connects to another
neuron in the next 2nd, or 3rd,or 4th,or … following
hidden layer. In this encoding scheme, the number of
repetition of that neuron depends on the number of
passed hidden layers, ñ, and is calculated as 2 ñ . It is
easy to realize that a chromosome such as abab bcbc,
unlike chromosome abab acbc for example, is not a
valid one in GS-GMDH networks and has to be
simply re-written as abbc .
a●
●ab
●abbc
b●
●bc
c●
However, in order to make it more general, it is
necessary to remove such restriction. In such
representation, neurons in different layers including
the input layer can be connected to others far away,
not only in the very adjacent layers. The encoding
schemes in generalized GMDH neural networks
must demonstrate the ability of representing different
length and size of such neural networks. Moreover,
the ability of changing building blocks of
information using crossover and mutation operators
must be preserved in such coding representation. In
the next sections, the encoding scheme of the newly
developed generalized GMDH neural networks is
discussed.
4.1 The Genome Representation of
Generalized GMDH Neural Networks
In the generalized GMDH neural networks, neurons
connections can occur between different layers
which are not necessarily very adjacent ones, unlike
the conventional structure GMDH neural networks in
which such connections only occur between adjacent
layers. For example, a network structure which
depicted in figure (1) shows such connection of
neuron ad directly to the output layer. Consequently,
this generalisation of network's structure can
evidently extend the performance of generalized
GMDH neural networks in modelling of real-world
complex processes. Such generalization is
accomplished by repeating the name of the neuron
which directly passing the next layers. In figure (1),
neuron ad in the first hidden layer is connected to the
output layer by directly going through the second
hidden layer. Therefore, it is now very easy to notice
that the name of output neuron ( network's output)
includes ad twice as abbcadad. In other words, a
virtual neuron named adad has been constructed in
●ad
●
abbcadad
d●
Figure 1: A Generalized GMDH network structure of a
chromosome
4.2 Genetic
Generalized
Reproduction
Operators
for
GMDH
Network
The genetic operators of crossover and mutation can
now be implemented to produce two offsprings from
two parents. The natural roulette wheel selection
method is used for choosing two parents producing
two offsprings. The crossover operator for two
selected individuals is simply accomplished by
exchanging the tails of two chromosomes from a
randomly chosen point as shown in figure (2). It
should be noted, however, such a point could only be
n 1
chosen randomly from the set { 21 , 2 2 , ..., 2 l }
where nl is the number of hidden layers of the
chromosome with the smaller length. It is very
evident from figures (2) and (3) that the crossover
operation can certainly exchange the building blocks
information of such generalized GMDH neural
networks so that the two type of generalized GMDH
and conventional GMDH-type neural networks can
be converted to
Parents
Offsprings
a b b c a d a d
a b b c
a b d d
a b d d a d a d
Figure 2: Crossover operation for two individuals in
generalized GMDH networks
each other, as can be seen from figure (3). In
addition, such crossover operation can also produce
different length of chromosomes which in turn leads
to different size of either generalized GMDH or
conventional GMDH network structures. Similarly,
the mutation operation can contribute effectively to
a
ab
b
bc
c
ad
ab
b
bc
abbc
abbcadad
c
a
ab
ab
abdd
b
b
c
abbc
d
d
a
a
c
abdd
abddadad
d
ad
d
Figure 3: Crossover operation on two generalized GMDH
networks
The incorporation of genetic algorithm into the
design of such GMDH-type neural networks starts
by representing each network as a string of
concatenated sub-strings of alphabetical digits. The
fitness, (  ), of each entire string of symbolic digits
which represents a GMDH-type neural network to
model the Pareto optimized data pairs is evaluated in
the form
Φ=1/E
,
(15)
where E, is the mean square of error given by
equation (9), is minimized through the evolutionary
process by maximizing the fitness  . The
evolutionary process starts by randomly generating
an initial population of symbolic strings each as a
candidate solution. Then, using the aforementioned
genetic operations of roulette wheel selection,
crossover, and mutation, the entire populations of
symbolic strings to improve gradually. In this way,
GMDH-type neural network models with
progressively increasing fitness, , are produced
until no further significant improvement is
achievable.
5 Multi-Objective Thermodynamic
Optimization and Modelling of
Turbojet Engines
Turbojet engines use air as the working fluid and
produce thrust based on the variation of kinetic
energy of burnt gases after combustion. The study of
thermodynamic cycle of a turbojet engine involves
different thermo-mechanical aspects such as
developed specific thrust, thermal and propulsive
efficiencies, and specific fuel consumption. A
detailed description of the thermodynamic analysis
and equations of ideal turbojet engines are given in
[10]. Input parameters involved in such
thermodynamic analysis in an ideal turbojet engine
given in Appendix A are Mach number (M0), input
air temperature (T0, K), specific heat ratio (  ),
heating value of fuel (hpr, kj/kg), exit burner total
temperature (Tt4, K), and pressure ratio, c. Output
parameters involved in the thermodynamic analysis
in the ideal turbojet engine are, specific thrust, (ST,
N/kg/sec), fuel-to-air ratio (f), specific fuel
consumption (SFC, kg/sec/N), thermal efficiency
(t), and propulsive efficiency (p). However, in
multi-objective optimization study, some input
parameters are already known or assumed as, T0 =
216.6 K,  =1.4, hpr =48000 kj/kg, and Tt4 = 1666 K.
The input Mach number 0 < M0 ≤ 1 and the
compressor pressure ratio 1 ≤ c ≤ 40 are considered
as design variables to be optimally found based on
multi-objective optimization of 4 output parameters,
namely, ST, SFC, ηt, and ηp. Consequently, four
objectives, namely, SFC, ST, p, and t, are chosen
for multi-objective optimization in which ST, p, and
t are maximized whilst SFC is minimized
simultaneously. A population size of 200 has been
chosen with crossover probability Pc and mutation
probability Pm as 0.8 and 0.02, respectively. The
result of such multi-objective optimization in the
plane of p and t is shown in figure (4). It should be
noted that such result could also be shown in
different plane; but such plane would be of the most
Training Data
Testing Data
0.45
0.4
Propulsive efficiency
the diversity of the population. This operation is
simply accomplished by changing one or more
symbolic digits as genes in a chromosome to another
possible symbols, for example, abbcadad to
abbccdad. It is very evident that mutation operation
can also convert a generalized GMDH network to a
conventional GMDH network or vice versa. It should
be noted that such evolutionary operations are
acceptable provided a valid chromosome is
produced. Otherwise, these operations are simply
repeated until a valid chromosome is constructed.
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0.6
0.62
0.64
0.66
0.68
0.7
0.72
Thermal efficiency
Figure 4: Non-dominated Pareto points in the plane of
efficiencies
important one in terms of modelling. Consequently,
the input-output data pairs used in the modelling
involve two different data tables which have been
just optimally found using multi-objective GAs for
thermodynamic Pareto optimisation of ideal turbojet
engines. The first table consists of four variables as
inputs, namely, SFC, ST, p, and t and one output
which is the Mach number. The second table consists
of the same four variables as inputs and another
output which is pressure ratio. It can be readily seen
that the results of a Pareto-based four-objective
optimization process for two design variables, Mach
number and pressure ratio, are inversely deployed for
a polynomial-type neural network modelling so that
the values of such design variables can now be
mathematically represented for the corresponding
combination values of four parameters, SFC, ST, p,
and t. In order to demonstrate the generalization
property of the evolved polynomial neural network,
data samples have been divided randomly for both
training and testing purposes, as shown in figure (4).
The unforeseen data samples during training process
or the test data have been merely used for testing
purpose. The structure of the evolved 1-hidden layer
GMDH-type neural network obtained for both design
variables is shown in figure (5).
(a)
(b)
Figure 5: Evolved GMDH-type neural networks for
modelling: (a) Mach No.
(b) Pressure ratio
The very good behaviours of such GMDH-type
network models are also depicted in scattered figure
(6) for both Mach number and pressure ratio,
respectively. It can be clearly seen that both models
have very good generalization ability in terms of
predicting unforeseen test data during training
process.
1.2
Computed & Predicted Mach No.
Computed & predicted pressure ratio
45
40
35
30
25
20
15
10
5
0
1
0.8
0.6
0.4
0.2
0
0
5
10
15
20
25
Actual pressure ratio
30
35
40
45
0
0.2
0.4
0.6
0.8
1
1.2
Actual Mach No.
(a)
(b)
Figure 6: Modelling behaviours of the evolved GMDH-type
neural networks in both training and testing data:
(a) Pressure ratio (b) Mach No.
Therefore, GMDH-type neural models shown in
figure (5) simply represent the optimal design
variables from a multi-objective optimisation point
of view for SFC, ST, p, and t.
6
Conclusion
Evolutionary methods for designing generalized
GMDH-type networks have been proposed and
successfully used for the modelling of input-output
data table obtained as the best Pareto front of
thermodynamic cycle of ideal turbojet engines. Such
Pareto front has been already found by a multiobjective optimization GA for the turbojet engine.
Such combination of modelling of non-dominated
Pareto optimized data pairs is very promising in
discovering useful
and interesting design
relationships.
References:
[1]Srinivas, N. and Deb, K., “ Multiobjective
optimization Using Nondominated Sorting in
Genetic Algorithms”, Evolutionary Computation,
Vol. 2, No. 3, pp 221-248, 1994.
[2]Coello Coello, C.A., “A comprehensive survey of
evolutionary based multiobjective optimization
techniques”, Knowledge and Information Systems:
An Int. Journal, (3), pp 269-308, 1999.
[3]Goldberg, D. E., “Genetic Algorithms in Search,
Optimization, and Machine Learning”, AddisonWesley, 1989.
[4]Deb, K., Agrawal, S., Pratap, A., Meyarivan, T.,
“A fast and elitist multi-objective genetic algorithm:
NSGA-II”, IEEE Trans. On Evolutionary
Computation 6(2):182-197, 2002.
[5] Astrom, K.J. and Eykhoff, P., “System
identification, a survey”, Automatica 7-123-62,
(1971).
[6] Ivakhnenko, A.G., “ Polynomial Theory of
Complex Systems”, IEEE Trans. Syst. Man &
Cybern, SMC-1, 364-378, (1971).
[7] Farlow, S.J., ed., “ Self-organizing Method in
Modeling: GMDH type algorithm”, Marcel Dekker
Inc., (1984).
[8] Nariman-Zadeh, N., Darvizeh, A., Ahmad-Zadeh,
R., “Hybrid Genetic Design of GMDH-Type Neural
Networks Using Singular Value Decomposition for
Modelling and Prediction of the Explosive Cutting
Process”, Proceedings of the I MECH E Part B
Journal of Engineering Manufacture, Vol. 217, pp.
779 – 790, (2003).
[9]Yao, X., “Evolving Artificial Neural Networks”,
Proceedings of IEEE, 87(9):1423-1447, Sept.,
(1999).
[10] Mattingly, J.P., “Elements of Gas Turbine
Propulsion”, Mc-Graw Hill, 1996