Optimizing Complex Systems
Sue Ellen Haupt
University of Nevada
Department of Mechanical Engineering
Reno, NV
702-784-6931
suehaupt@moriah.unr.edu
Randy L. Haupt
University of Nevada
Electrical Engineering Departmend260
Reno, NV 89557-0153
702-784-6927
haupt 0ee.unr.edu
Abstract- Modeling and solving complex engineering
problems requires new algorithmic tools. A promising new
tool for numerical optimization is the genetic algorithm.
This paper presents two interesting applications using the
genetic algorithm. The first optimizes a function that has a
subjective output: music. The second uses continuous
parameters rather than the traditional binary parameters to
find solutions to a nonlinear partial differential equation. A
brief introduction to the continuous parameter genetic
algorithm is given.
TABLEOF CONTENTS
1.
2.
3.
4.
INTRODUCTION
A SUBJECTIVE COST FUNCTION
CONTINUOUS
PARAMETER
GENETICALGORITHM
THE SUPER KORTEWEGDE VRIES EQUATION
5. RESULTS
6 . CONCLUSIONS
1. INTICODUCTION
Our ability to model complex systems improves as
computational resources get more powerful. Traditional
approaches to finding solutions prove inadequate in many
cases. For instance, numerical optimization has relied upon
either random sampling of the output or some form of a
calculus-based method of finding the minimum. Random
sampling proves to be extremely time-consuming, and
calculus-based approaches lhave narrow search regions.
Mother Nature runs the most complex experiments today
and has some interesting approaches to optimization. The
genetic algorithm is a type of evolutionary algorithm that is
modeled on biological processes and can be used to
optimize highly complex cost functions. A genetic algorithm
attempts to cause a population composed of many
individuals or potential solutions to evolve under specified
selection rules to a state which finds the “most fit”
individuals (i.e. minimizes the cost function, assuming it has
been written so that the minimum value is the desired
solution). The method was developed by John Holland [ l ]
0-7803-431 1-5/98/$10.00~ 0 1998 IEEE
over the course of the 1960’s and 1970s and finally
popularized by one of his students, David Goldberg, who
was able to solve a difficult problem involving the control of
gas-pipeline transmissioin for his dissertation [2]. Several
introductory papers on genetic algorithms were presented at
this conference in 1996 [3,4], so we refer to those papers for
a discussion on the veIy basics. The parameters of the
problem that are to be optimized are coded into arrays
known as chromosomes. Each parameter is a gene. The GA
begins with a large population of chromosomes. Each
chromosome is evaluated by the cost, function for the
problem and assigned a cost. The most fit chromosomes
with the lowest costs suirvive to mate. The mating process
combines information from the two parent chromosomes to
form offspring, which jioin the population. The process
iterates. Along the wa:y, some chromosomes mutate or
experience a change in some of their genes. The process
continues until convergence criteria are met. Choosing the
best population size, mating scheme, and mutation rate are
till more of an art.
Some of the advantages of a genetic algorithm include that it
Optimizes with continuous or discrete parameters,
Doesn’t require derivative information,
Simultaneously searches from a wide sampling of the
cost surface,
Deals with a large number of parameters,
Is well suited for parallel computers,
Optimizes parameters with extremely complex cost
surfaces,
Provides a list of semi-optimum parameters, not just a
single solution,
May encode the parameters so that the optimization is
done with the encoded parameters, and
Works with numerically generated data, experimental
data, or analytical functions.
These advantages are intriguing and often produce stunning
results when traditionlal optimization approaches fail
miserably.
24 1
The large population of solutions that gives the genetic
algorithm its power, is also its bane when it comes to speed
on a serial computer--the cost function of each of those
solutions must be evaluated. However, if a parallel computer
is available, each processor can perform a separate function
evaluation at the same time. For a GA, the function
evaluations are generally decoupled from the rest of the
routine and particularly easy to parallelize. Thus, the
genetic algorithm is optimally suited for such parallel
computations.
In this case, we know the correct answer, and the notes are
shown in Figure 1. Putting the song parameters into a row
vector yields
This paper introduces two examples of optimizing complex
systems. The first example uses a binary genetic algorithm
to find the notes to a song. Two types of cost functions are
presented: objective and subjective. This example lays the
foundation for a method to optimize extremely complex
systems where the computer and engineer work as a team.
The second example uses the less familiar continuous
parameter genetic algorithm to find equilibrium solutions to
a complex nonlinear equation. A short introduction to the
continuous genetic algorithm is given.
We’ll attempt to find this solution two ways. First, the
computer compares a chromosome with the known answer
and gives a point for each bit in the proper location. Second,
we’ll use our ear (very subjective) to rank the chromosomes.
2. A SUBJECTIVE COST FUNCTION
In the first example the musical “talent” of the genetic
algorithm is tested to see if it could learn the melody of the
first four measures of “Mary Had a Little Lamb” [5].This
song is in the key of C with 414 time and has only quarter
and half notes. In addition, the frequency variation of the
notes is less than octave. A chromosome for this problem
has 4 x 4 = 16 genes (one gene for each beat). The binary
genetic algorithm is perfect, because there are eight distinct
notes, seven possible frequencies and one hold (the
encoding is given in Table 1). A hold indicates the previous
note is a half note. It is possible to devote one bit to indicate
a quarter or half note. However, this approach would require
variable length chromosomes. The actual solution only
makes use of the C, D, E, and G notes and the hold.
Therefore, the binary encoding is only 62.5% efficient.
Since the exact notes in the song may not be known, the
inefficiency is only evident in hindsight. An exhaustive
search would have to try 816 = 2 . 8 1 4 7 ~ 1 0 ’ ~
possible
combinations of notes.
[EDCDEEEholdDDDholdEGGhold]
with a corresponding chromosome encoded as
chromosome=[lOl 100 011 100 101 101 101 000 100 100
100000101 111 111000].
Figure 1. Music to “Mary Had a Little Lamb.”
The first cost function subtracts the binary guess (guess)
from the true answer (answer) and sums the absolute value
of all the digits. When a chromosome and the true answer
match, the cost is zero.
48
n=l
The genetic algorithm has a population of 48, 24
chromosomes are discarded each iteration, and a mutation
rate of 5%. Figure 2 shows an example of the cost function
statistics as a function of generation. The genetic algorithm
consistently finds the correct answer over many different
runs.
L
’
O’
5
Table 1. Binary encoding of the musical notes for “Mary
Had a Little Lamb.”
I
I
242
100
111
I
I
D
G
]
]
10
15
20
25
generation
30
35
40
Figure 2 . The minimum cost and mean cost as a function of
generation when the computer knows the exact answer (all
the notes to “Mary Had a Little Lamb”).
The second cost function is subjective. This is an interesting
twist in that it combines the computer with a human
response to judge performance. Cost is assigned from a 0
(that’s the song) to a 100 ((terrible match). This algorithm
wears on your nerves, since some collection of notes must
be played and judged for e;ach chromosome. Consequently,
the authors could only listen to the first two measures and
judge the performance. Figure 3 shows the cost function
statistics as a function of generation. Compare this graph
with the previous one.
Subjective cost functions are quite fascinating. A
chromosome in the first few generations tended to receive a
lower cost than the id’entical chromosome in later
generations. This accounts for the increase in the minimum
cost shown in Figure 3. Miany chromosomes produce very
unmeiodic sounds. Raising expectations or standards is
common in everyday life. For instance, someone who begins
a regular running program is probably happier with a tenminute mile, than she would be after one year of training.
The subjective cost functnon converged faster than the
mathematical cost function, because a human is able to
evaluate more aspects of the experiment than the computer.
Consider the difference between a G note and the C, E, and
F notes. The human ear rartks the three notes according to
how close they are to G, while the computer only notices the
three notes differ from G by one bit. A continuous parameter
genetic algorithm might work better in this situation.
“VI
---__
~
I
I
---- _ _ _ _ _ _
,A\.
t
10
wl
2
3
4
5
Most companies use computers to design products. Various
parameters, bounds, and tolerances are input to the computer
and eventually a design pops out. Perhaps a better approach
is to have a closer interaction between the computer and the
designer. A computer calculates well and does mundane
tasks such as sorting a list well, but it is not good at making
subjective judgments. If the computer program occasionally
stops and presents some options and their associated costs to
the designer, then the designer has the ability to add a
human element to the design. Since the genetic algorithm
carries a list of answers in descending order of preference, it
is perfect for fuzzy desigri applications.
3. CONTINUOUS
PARAMETER
GENETIC
ALGORITHM
---_
30I
t
“The primary problems with these programs seem to be
those involving the human interface. The time it takes to run
the program and listen to the samples combined with the
relative dullness of the samples mean that the users rapidly
get tired and bored. This means that the programs need to be
built to generate the samlples relatively quickly, the samples
need to be relatively short, the samples need to develop and
evolve to be interesting quickly and of necessity the
population size must be small.”
mean cost
minimum cost
*O
mathematical functions (like sine and cosine) in the time
domain. This approach produced too much noise, and
people had difficulty judging the performance (patience and
tolerance were difficult). The second approach used notes as
in our example. This approach worked better, but listening
to many randomly generated music pieces is too difficult for
most of us. He concluded:
6
7
8
generation
Figure 3. The minimum cost and mean cost as a function of
generation when the subjective cost function was used (our
judgment as to how close the song associated with a
chromosome came to the actual tune of “Mary Had a Little
Lamb”).
Extensions to this idea are Ilimitless. Composing music [6],
[7], [8] and producing art with a genetic algorithm are
interesting possibilities. Putrnan tried his hand at composing
music with a genetic alglorithm [7]. The idea was to
construct random pieces of music and rate them for their
musical value. He used two approaches. The first combined
Our second example makes use of the continuous parameter
genetic algorithm, so we give a short introduction here. The
flow chart in Figure 4 prowides a “big picture” overview of a
continuous genetic algorithm. Like the binary genetic
algorithm, it begins %with a random population of
chromosomes which represent the encoded parameters. The
difference is that these parameters are real numbers. These
chromosome strings are fed to the cost function for
evaluation. The fittest chromosomes survive and the highest
cost ones die off. This process mimics natural selection in
the natural world. The lowest cost survivors mate. The
mating process combines information from the two parents
to produce two offspring. Some of the population
experiences mutations. The mating and mutation operators
introduce new chromosoines which may have a lower cost
than the prior generation. The process iterates until an
acceptable solution is fouind.
This algorithm doesn’t require explicit conversions between
continuous and binary parameters. In addition, the mating
and mutation functions are significantly different from the
binary version. Consequently, we explain the new mating or
crossover functions and continuous parameter mutations.
243
Crossover
Once two parents are chosen, some method must be devised
to produce offspring which are some combination of these
parents. Many different approaches have been tried for
crossing over in continuous parameter genetic algorithms.
Adewuya [9] reviews some of the current methods
thoroughly. Several interesting methods are demonstrated by
Michalewicz [ 101.
Our algorithm is a combination of an extrapolation method
with a crossover method. We wanted to find a way to closely
mimic the advantages of the binary genetic algorithm mating
scheme. It begins by randomly selecting a parameter in the
first pair of parents to be the crossover point
a = roundup( random x N p a r }
I
~
Select Initial Population
of Continuous Pikameters
(2)
I
1
where p is also a random value between 0 and 1. The final
step is to complete the crossover with the rest of the
chromosome as before:
If the first parameter of the chromosomes is selected, then
only the parameters to right of the selected parameter are
swapped. If the last parameter of the chromosomes is
selected, then only the parmeters to the left of the selected
parameter are swapped. This method does not allow
offspring parameters outside the bounds set by the parent
unless p is greater than one.
Mutations
If care is not taken, the genetic algorithm converges too
quickly into one region of the cost surface. If this area is in
the region of the global minimum, that is good. However,
some functions, such as the one we are modeling, have many
local minima. If we do nothing to solve this tendency to
converge quickly, we could end up in a local rather than a
global minimum. To avoid this problem of overly fast
convergence, we force the routine to explore other areas of
the cost surface by randomly introducing changes, or
mutations, in some of the parameters. A mutated parameter
is replaced by a new random parameter.
1
Natural Selection I
4.THESUPER KORTEWEG
DE VRES EQUATION
done
Figure 4. Flow chart of a continuous genetic algorithm.
We'll let
where the m and d subscripts discriminate between the mom
and the dad parent. Then the selected parameters are
combined to form new parameters that will appear in the
children:
244
Some of the modeling tools of scientists and engineers are
differential and partial differential equations (PDEs).
Solution methods typically depend on the specifics of the
equation being solved. When the equation is nonlinear,
finding a solution suddenly becomes extremely difficult.
Although a few nonlinear PDEs can be solved analytically,
we often must rely on numerical methods to determine an
approximate solution. One class of PDEs of particular
interest to engineers and scientists is the solitary wave.
Solitary waves, or solitons, are permanent form waves for
which the nonlinearity balances the dispersion to produce a
coherent structure. They appear as models of many coherent
phenomena ranging from propagation patterns in optical
cables to the Great Red Spot of Jupiter. Here, we are
interested in demonstrating that a genetic algorithm is a
useful technique for solving a highly nonlinear differential
equation that is formally nonintegrable.
The equation to be solved here the Super Korteweg de Vries
(SKDV) Equation:
ut = m u ,
-+ pu*
-vu,,
=0
(5)
The functional form is denoted by u , time derivative by the
t subscript, spatial derivative by the x subscript, and a , p ,
and u are parameters of the problem. It is a model for
several physical phenomenal, including shallow water waves
near a critical value of surface tension, magneto-acoustic
waves propagating at an angle to an external magnetic field,
and waves in a nonlinear LC circuit with mutual inductance
between neighboring inductors [ 111. We wish to solve for
waves that are steadily translating so we write the t variation
using a Galilean transformation, X=x-ct where c is the phase
speed of the wave. Thus, our SKDV becomes a fifth order,
nonlinear ordinary different:ialequation:
(7)
k=l
Without loss of generality, we have assumed that the
function is symmetric about the X-axis by not including sine
functions. In addition, we use the “cnoidal convention”’ by
assuming that the constant is 0. Now, we can easily take
derivatives as powers of the wave numbers to write the cost
that we wish to minimize as:
k=l
5. RESULTS
Boyd [I21 extensively studied methods of solving this
equation. He expanded the solution in terms of Fourier
series to find periodic cnoiidal wave solutions (solitons that
are repeated periodically). Among the methods used are the
analytical Stokes’ expansion, which intrinsically assumes
small amplitude waves, and the numerical NewtonKantorovich iterative method, which can go beyond the
small amplitude regime if care is taken to provide a very
good first guess. This m.ethod is based on the Newton
iteration and requires an analytical expansion of the equation
into a mean plus a perturbation. The equation is solved for
the perturbation. The equation is solved for the perturbation
and then discretized. At each step of the iteration, the
perturbation is found and added to the prior solution. Such
method assumes that the pe.rturbation is small, thus requiring
a good “first guess” to the solution.
Haupt and Boyd [I31 were: able to extend these methods to
deal with resonance conditions. These methods were
generalized to two dimensions to find double cnoidal waves
(two waves of differing wave number on each period) for the
integrable Korteweg de Vries equation [I41 and the
nonintegrable Regularized, Long Wave Equation [ 151.
However, these methods require careful analytics and
programming that is very problem-specific. Here, we use a
genetic algorithm to obtain a similar result.
Normally, we don’t think of PDEs as minimization
problems. However, if we want to find for which values a
differential equation is zero (a form in which we can always
cast the system), we can look for the minimum of its
absolute value. Koza [I61 demonstrated that a genetic
algorithm could solve a simple differential equation by
minimizing the value of the solution at 200 points. To do
this, he numerically differentiated at each point and fit the
appropriate solution. For this fifth order equation, this is not
so easy. Therefore we combine the genetic algorithm with a
spectral expansion which eases the differentiation.
To find the solution of (ti), we expand the function u in
terms of a Fourier cosine series:
Equation (8) is reasonablly easy to put into the cost function
of a continuous genetic: algorithm. We wish to find the
coefficients of the series, a k . Thus, the ak are the
continuous parameters encoded into the chromosome of the
genetic algorithm.
The only minor complication is
computing u in the nonlinear term to insert into the cost
function (8). However, this is merely one extra line of code.
We just use the spectral expansion of the current
chromosome.
The parameters that we used here are v = 1, p = 0, a = 1,
and a phase speed of ,c=14.683 to match with a known
highly nonlinear solution. Note that the phase speed and
amplitude of solitary type waves are interdependent. We
could instead have specified the amplitude and solved for
the phase speed. It is equivalent. We computed the
coefficients, a k , to find the best cnoidal wave solution for
K=6. We used N i p , = 500, N , ,
= 100, mutation rate of
0.2, and 70 iterations. We evaluated the cost function at
merely 2 points for this run and summed their absolute
value. The results appear in Figure 5 . The solid line is the
“exact“ solution reported by Boyd [12] and the dashed line
is the genetic algorithm’s approximation to it. They are
barely distinguishable. For interest’s sake we show a genetic
algorithm solution that converged to a double cnoidal wave
as Figure 6 . These double cnoidal wave solutions would
require a two-dimensional expansion involving two different
phase speeds in the Newton-Kantorovich method. Here, we
were able to find them using the same basic cost function as
for the single cnoidal wave. We are currently working on
modifying the cost function to specify which form we wish
to find a priori.
6 . CONCLUSIONS
Genetic algorithms show promise for optimization problems
in complex systems. Htere we have given two examples of
problems where they may prove useful in the future. In the
first problem, the binary genetic algoritm was used to
245
-40
1
-4
I
-3
I
I
-2
-1
I
I
I
I
0
1
2
3
4
X
Figure 5. The cnoidal wave of the Super Korteweg de Vries equation. Solid line - exact solution. Dashed line - the genetic
algorithm solution.
'
-1 00
-4
1
-3
I
-2
I
-1
I
I
I
I
0
1
2
3
4
X
Figure 6. A double cnoidal wave of the Super Korteweg de Vries equation as computed by the genetic algorithm.
246
compose music, both with a1 well-defined cost function to a
known solution and also to ,a subjective cost function. This
example demonstrated how a numerical algorithm can be
interfaced to human judgement to produce an artistic result.
The second example was much more arcane and involved
finding equilibrium solutions to a highly nonlinear fifth
order differential equation.
[ 131 S.E. Haupt and J.P.Boyd, “Modeling Nonlinear
Resonance: A Modification to the Stokes‘ Perturbation
Expansion,” Wave Motion, 10, 1988, pp. 83-98
REFEIRENCES
[15] S.E. Haupt, “Solvinlg Nonlinear Wave Problems with
Spectral Boundary Value Techniques”, Ph.D. Dissertation,
University of Michigan, 1988, 157pp.
[l] J. H. Holland, “Genetic ;algorithms,” Sci. Amer., pp. 6672, July 1992.
[2] D.E. Goldberg, Genetic Algorithms in Search,
Optimization, and Machine Leaming, New York AddisonWesley, 1989.
[3] J.M. Johnson and Y. Rahmat-Sami, “Genetic algorithm
optimization for aerospace electromagnetic design and
analysis,” 1996 IEEE Aerospace Applications Con$,
Snowmuss, CO, Feb 3-10, pp. 87-102.
[4] R.L. Haupt, “Genetic algorithm design of antenna
arrays,” I996 IEEE Aerospace Applications Con$,
Snowmass, CO, Feb 3-10, pp. 103-109.
[5] R.L. Haupt and S.IE. Haupt, Practical Genetic
Algorithms, New York: John Wiley & Sons, 1997.
[6] B. L. Jacob, “Composing with genetic algorithms,”
Proceedings of the International Computer Music
Conference, September 1995.
[7] J. B. Putnam, “Genetic Programming of Music,” New
Mexico Institute of Mining and Technology, Socorro, NM,
30 Aug 94.
[8] J. A. Biles, “GenJam: A genetic algorithm for generating
jazz solos,” Proceedings of the International Computer
Music Conference, 1994.
[9] A.A.Adewuya, “New Methods in Genetic Search with
Real-valued Chromosomes,” Master’s Thesis, Cambridge,
Massachusetts Institute of Technology, 1996.
[ 101 Z. Michalewicz, Genetic Algorithms
+ Data Structures
= Evolution Programs, New York: Springer-Verlag, 1992.
[ 111 K . Yoshimura and S. ‘Watanabe, “Chaotic behavior of
1141 S.E. Haupt and J.P. Boyd, “Double Cnoidal Waves of
the Korteweg De Vries Equation: Solution by the Spectral
Boundary Value Approach,” Physica D, 50, 1988, 117-134.
[16] J.R. Koza, “Chapter 10: The Genetic Programming
Paradigm: Genetically Breeding Populations of Computer
Programs to Solve Problems,” in Dynamic, Genetic, and
Chaotic Programming, The Sixth Generation, B. Soucek,
ed., New York John Wiley & Sons, Inc., 1992, pp.203-321.
Sue Ellen Haupt is an Associate
Research Professor of Mechanical
Engineering at the University of
Nevada, Reno. Dr. H ~ p received
t
her Ph.D. in Atmospheric Science
from the University of Michigan, her
MSME from Worcester Polytechnic
Institute,
MS
Engineering
Management from Western New
England College, and B.S. in Meteorology from The
Pennsylvania State University. She has spent much of her
career with the National Center for Atmospheric Research
and the University of Colorado/ Boulder. Her research
encompasses fluid flow, numerical methods, large-scale
geophysical fluid dynamics and dynamical systems.
Randy L Haupt is Professor and
Chair of Electrical Engineering at
the University of Nevada at Reno.
He recently retired from! the USAF
as a Lieutenant Colonel and
Professor of Electrical Engineering
at the USAF Academy, CO. His work
experience
includes
electrical
engineer for the OTH-B Radar
Program Oflce and an antenna research engineer for Rome
Air Development Center, both at Hanscom AFB, MA. He
received his Ph.D. from the University of Michigan, his
MSEE from Northeastern University, MS in Engineering
Management for Westem New England College, and BSEE
from the USAF Academy.
nonlinear evolution equation with fifth order dispersion,” J.
Phys. SOC.Japan, 5 1 , 1982, lpp. 3028-3035.
[ 121 J.P. Boyd, “Solitons from Sine Waves: Analytical and
Numerical Methods for Nonintegrable Solitary and Cnoidal
Waves,” Physica 21D, 1986.,pp. 227-246.
247