What is Genetic Algorithm?
A genetic algorithm is an optimization method which based on natural selection and is simple to
implement.
The concept
Inspired by Darwin’s Theory of evolution popularly termed as “Survival of the fittest”. The fitness is
maintained through the process of reproduction, crossover and mutation.
Development
The idea behind dates back to the days of Alan Turing who had proposed a “learning machine” which
would work in the lines of the evolution process. In 1960s the idea of evolution strategy/programming
was developed in Germany and in the USA. However, Genetic Algorithm became popular through the
works of John Holland and his students in the early 1970s (book: "Adaption in Natural and Artificial
Systems"(1975)). David Goldberg, who was a graduate student of Holland’s, presented an awardwinning doctoral thesis on his GA application to gas pipeline optimization. In 1989, Goldberg also
wrote an influential book—Genetic Algorithms in Search, Optimization, and Machine Learning. This
was the final catalyst in setting off a sustained development of GA theory and applications that is still
growing rapidly.
The biological connection:
http://www.ai-junkie.com/ga/intro/gat1.html
Every organism has a set of rules, a blueprint so to speak, describing how that organism is built up
from the tiny building blocks of life. These rules are encoded in the genes of an organism, which in
turn are connected together into long strings called chromosomes. Each gene represents a specific
trait of the organism, like eye colour or hair colour, and has several different settings. For example,
the settings for a hair colour gene may be blonde, black or auburn. These genes and their settings are
usually referred to as an organism's genotype. The physical expression of the genotype - the organism
itself - is called the phenotype.
When two organisms mate they share their genes. The resultant offspring may end up having half the
genes from one parent and half from the other. This process is called recombination. Very occasionally
a gene may be mutated. Normally this mutated gene will not affect the development of the phenotype
but very occasionally it will be expressed in the organism as a completely new trait.
Living organisms follow the processes of natural selection - survival of the fittest - and gene mutation
have very powerful roles to play in the evolution of an organism.
Genetic Algorithms are a way of solving problems by mimicking the same processes Mother Nature
uses. GA works along that line in the sense that it tries to find the best combination from the parent
population to generate a fit gene through crossover and then eventually mutate and progress
through several generations until the fittest/ optimal solution is reached. It is a randomization
process and usually leads to the global optimized solution.
Concepts
If we are solving some problem, we are usually looking for some solution, which will be the best
among others. The space of all feasible solutions (it means objects among those the desired solution
is) is called search space (also state space). Each point in the search space represent one feasible
solution. Each feasible solution can be "marked" by its value or fitness for the problem. We are
looking for our solution, which is one point (or more) among feasible solutions - that is one point in
the search space.
1
The looking for a solution is then equal to a looking for some extreme (minimum or maximum) in the
search space. The search space can be whole known by the time of solving a problem, but usually we
know only a few points from it and we are generating other points as the process of finding solution
continues.
The problem is that the search can be very complicated. One does not know where to look for the
solution and where to start. There are many methods, how to find some suitable solution (ie. not
necessarily the best solution), for example hill climbing, tabu search, simulated annealing and genetic
algorithm. The solution found by this methods is often considered as a good solution, because it is
not often possible to prove what the real optimum is.
A GA is a stochastic technique with simple operations based on the theory of natural selection. The
basic operations are selection of population members for the next generation, ``mating'' these
members via crossover of ``chromosomes,'' and performing mutations on the chromosomes to
preserve population diversity so as to avoid convergence to local optima. Finally, the fitness of each
member in the new generation is determined using an evaluation (fitness) function. This fitness
influences the selection process for the next generation.
The GA operations selection, crossover and mutation primarily involve random number generation,
copying, and partial string exchange. Thus they are powerful tools which are simple to implement. Its
basis in natural selection allows a GA to employ a ``survival of the fittest'' strategy when searching for
optima. The use of a population of points helps the GA avoid converging to false peaks (local optima)
in the search space.
Why GA?
http://www.wardsystems.com/ga101.asp
Exhaustive search (where every possible combination is checked) was used extensively for
optimization problems and it gave an accurate result. However, it is a very inefficient approach
because when the numbers of variables are large this process is time consuming. That is why users of
exhaustive search optimizers tend to limit the number of variables they use, or tend to limit the
number of values these variables can take.
The genetic algorithm, by contrast, does not try every possible combination. It attempts to reach the
optimal solution by selecting variables intelligently. Therefore, far more variables can be utilized,
and all values of a variable can be checked.
Genetic algorithms are searches different areas of the search space simultaneously. As a result they
are less likely to become stuck in "local minima" as the others quite often do. (Local minima are
decent solutions that the optimizer can never get out of in order to find better solutions.)
Outline of the Basic Genetic Algorithm
[Start] Generate random population of n chromosomes (suitable solutions for the problem)
[Fitness] Evaluate the fitness f(x) of each chromosome x in the population
[New population] Create a new population by repeating following steps until the new population is
complete
[Selection] Select two parent chromosomes from a population according to their fitness (the better
fitness, the bigger chance to be selected)
[Crossover] With a crossover probability cross over the parents to form a new offspring (children). If
no crossover was performed, offspring is an exact copy of parents.
[Mutation] With a mutation probability mutate new offspring at each locus (position in chromosome).
2
[Accepting] Place new offspring in a new population
[Replace] Use new generated population for a further run of algorithm
[Test] If the end condition is satisfied, stop, and return the best solution in current population
[Loop] Go to step 2
Differences between GA and other conventional methods
A genetic algorithm (GA) is a natural selection-based optimization technique. There are four major
differences between GA-based approaches and conventional problem-solving methods:
 GAs work with a coding of the parameter set, not the parameters themselves
 GAs search for optima from a population of points, not a single point
 GAs use payoff (objective function) information, not other auxiliary knowledge such as
derivative information used in calculus-based methods;
 GAs use probabilistic transition rules, not deterministic rules. These four properties make GAs
robust, powerful, and data-independent.
When are Genetic Algorithms Useful?
There are at least three situations where genetic algorithms are useful:
 The objective function is not smooth (i.e., not differentiable).
 There are multiple local optima.
 There is a large number of parameters (the meaning of “large” keeps changing).
If the objective function does not have a derivative, then clearly a derivative-based algorithm cannot
be used. There are several choices of algorithm for a problem like this — genetic algorithms are one.
One way to have an objective that is not differentiable is if one or more of the parameters is integervalued.
Derivative-based algorithms take no allowance of multiple optima — they go to a local optimum near
to where they start. If there are likely to be only a few local optima, then using several random starts
may be enough to solve the problem.
Derivative-based algorithms often progress slowly when there is a large number of parameters. If the
gradient is being evaluated numerically, then each iteration of the optimization requires as many
function evaluations as there are parameters. The space to be searched is vast, so a large number of
iterations will probably be required. In general, derivative-based algorithms work better as they
approach the optimum. So a reasonable strategy for many problems with a lot of parameters is to
start with a genetic algorithm, then use the best solution from the genetic algorithm as the start for
a derivative-based optimizer.
GAs are used in
Genetic algorithms have been applied to many hard optimization problems.
 VLSI layout optimization
 Boolean satisfiability
 the Hamiltonian circuit problem
 model selection specially when the objective function is non linear
It can solve every optimization problem which can be described with the chromosome encoding.
3




It solves problems with multiple solutions.
Since the genetic algorithm execution technique is not dependent on the error surface, we can
solve multi-dimensional, non-differential, non-continuous, and even non-parametrical problems.
Structural genetic algorithm gives us the possibility to solve the solution structure and solution
parameter problems at the same time by means of genetic algorithm.
Genetic algorithm is a method which is very easy to understand and it practically does not demand
the knowledge of mathematics.
 Genetic algorithms are easily transferred to existing simulations and models.
Drawbacks:
 When the exact fitness function is not known/ poorly known then certain optimization problems
(they are called variant problems) cannot be solved by means of genetic algorithms.
 There is no absolute assurance that a genetic algorithm will find a global optimum. It happens very
often when the populations have a lot of subjects.
 The optimization response time is not constant. As such it limits the use of GA in real time
application. Even more, the difference between the shortest and the longest optimization
response time is much larger than with conventional gradient methods.
 Genetic algorithm applications in controls which are performed in real time are limited because
of random solutions and convergence, in other words this means that the entire population is
improving, but this could not be said for an individual within this population. Therefore, it is
unreasonable to use genetic algorithms for on-line controls in real systems without testing them
first on a simulation model.
References
References and further readings:
 Denny Hermawanto. Genetic Algorithm for Solving Simple Mathematical Equality Problem.
 Hillier and Lieberman. Introduction to Operations Research (9th Ed.)
 Hosmer, D.W. and Lemeshow, S. Applied Logistic Regression. New York: Wiley
 Givens & Hoeting.Computational Statistics (2nd Ed.)
 K.F Man, et all. Genetic Algorithms: Concepts and Applications. IEEE Transactions on Industrial
Electronics, Vol.43 No 5, Oct 1996
 Luca Scrucca. GA: A Package for Genetic Algorithms in R. Journal of Statistical Software, Vol 23
, Issue 4, April 2013
 V. Calcagno, et all. glmulti: An R Package for Easy Automated Model Selection with
(Generalized) Linear Models. Journal of Statistical Software, Vol 24 Issue 12, May 2010.
 http://www.statsblogs.com/2012/08/01/genetic-algorithms-a-simple-r-example/ (same
problem as the R-bloggers site)
 http://www.obitko.com/tutorials/genetic-algorithms/
 http://www.burns-stat.com/documents/tutorials/an-introduction-to-genetic-algorithms/
 http://lancet.mit.edu/ga/dist/
4
R codes used in the presentation:
Example 1:
babywt=read.table(file="C:\\Users\\dola\\Desktop\\SPRING 2015\\STAT
992\\Presentation\\smallbabies_data.txt",sep="",header=TRUE)
lbw <- within(babywt, {
## race relabeling
race.cat
<factor(race,
levels
=
1:3,
labels
=
c("White","Black","Other"))
## ftv (frequency of visit) relabeling
ftv.cat <- cut(ftv, breaks = c(-Inf, 0, 2, Inf), labels =
c("None","Normal","Many"))
ftv.cat <- relevel(ftv.cat, ref = "Normal")
## ptl
preterm <- factor(ptl >= 1, levels = c(F,T), labels = c("0","1+"))
##bwt in pounds
bwt<-babywt$bwt_gram/2200
})
head(lbw)
# The fitness function:
#est_bwt<-1.0116+0.0033*x+0.0019*y+0.007*z
#x= age , 14<x<45
#y= weight of the mother in lbs, 80<y<250
#z= ftv, 0<z<6
#generating the initial population
#having four individuals/ chromosomes each having three
#genes (age, weight, ftv)
fit<-function(chrom)
{
ft<-matrix(NA, 1,4)
for(i in 1:4){
ft[i]<1.0116+0.0033*chrom[1,i]+0.0019*chrom[2,i]+0.007*chrom[3,i
]
}
ft
}
#STEP1:Generate random population of 4 chromosomes
set.seed(7777)
x<-floor(runif(4, 14,45))
y<-floor(runif(4, 80,250))
z<-floor(runif(4, 0,6))
5
population_chrom<-matrix(c(x,y,z),byrow
=
TRUE,nrow=3,ncol=4) #population units
population_chrom
C1<-population_chrom[,1]
C2<-population_chrom[,2]
C3<-population_chrom[,3]
C4<-population_chrom[,4]
#STEP2 Evaluate the fitnessf(x) of each chromosome in the
#population using the fitness function
# est lbw<-1.3051+0.5*x+0.0018*y+3*z
fitness<-fit(population_chrom)
#STEP 3:Select two parent chromosomes from a population
#according to their fitness and comparing the probabilities
#with randomly generated probabilities to check for
#reordering
objective<-fitness
total<-sum(objective)
#probabilities associated with each chromosome
prob<-objective/total
set.seed(9001)
gen_prob<-runif(4, min=0, max=1)
data.frame(t(prob), gen_prob)
# the rule: if the generated probability is less than the
#true probabilities then that chromosome is not replaced.
#If the generated probability is larger than the
#corresponding true probabilites then the chromosome is
#reordered with the next chromosome where generated
#prob< true prob
#NOTE :if generated prob< true prob for none of the
#chromosome then the last chromosome is used generating the
#new order of chromosome for the 1st generation
newC1<-C2
newC2<-C1
newC3<-C3
newC4<-C4
# reordered chromosome
Chrom_reordered<matrix(c(newC1,newC2,newC3,newC4),nrow=3,ncol=4)
Chrom_reordered
6
#STEP 4:CROSSOVER
NewChrom<-matrix(NA,nrow=3, ncol=4)
rho<-0.35
#crossover probabilty
R<-matrix(NA,nrow=4, ncol=1)
#checking for the chromosomes which will undergo crossover
set.seed(7755)
for(k in 1:4){
R[k]<-runif(1, min=0, max=1)
R[k]
}
R
#compare the R values with rho and those chrosomosomes whose
#R values are less than rho will undergo crossover
#Deciding the position on which the crossover will take
#place
set.seed(6655)
cross<-floor(runif(1, min=1, max=2))
cross
new_c1<-newC1
new_c2<-newC2
new_c3<-c(33,196,1)
new_c4<-c(44,229,5)
#The population after the crossover
Chromosome_cross<-matrix(c(new_c1,
new_c2,
new_c3,new_c4),nrow=3,ncol=4)
Chromosome_cross
# STEP 5 MUTATION
tot_len_gene<-3*4
mut<-0.20 #mutation probability
mut_gen<-floor(tot_len_gene*mut)
needs to be mutated
set.seed(7761)
#genes to be mutuated
floor(runif(2, min=1, max=12))
#for position 5: variable lwt
floor(runif(1, min=80, max=250))
# for position 4: variable Age
floor(runif(1, min=14, max=45))
#number
of
genes
that
7
#mutating the gene
Chromosome_cross[2,2]<-235
Chromosome_cross[1,2]<-29
# 1st generation
first_gen<-Chromosome_cross
first_gen
#evaluated fitness values for the first generation
fitness_new<-fit(first_gen)
data.frame(t(fitness),t(fitness_new))
Example 2:
babywt=read.table(file="C:\\Users\\dola\\Desktop\\SPRING 2015\\STAT
992\\Presentation\\smallbabies_data.txt",sep="",header=TRUE)
head(babywt)
lbw <- within(babywt, {
## race relabeling
race.cat
<factor(race,
levels
=
1:3,
labels
=
c("White","Black","Other"))
## ftv (frequency of visit) relabeling
ftv.cat <- cut(ftv, breaks = c(-Inf, 0, 2, Inf), labels =
c("None","Normal","Many"))
ftv.cat <- relevel(ftv.cat, ref = "Normal")
## ptl
preterm <- factor(ptl >= 1, levels = c(F,T), labels = c("0","1+"))
##bwt in pounds
bwt<-babywt$bwt_gram/2200
})
head(lbw)
library(glmulti)
###########################
# Exhaustive with no interaction
###########################
start.time<-proc.time()
glmulti.lm.out <glmulti(bwt ~ age + lwt + race.cat + sm + preterm + ht + ui +
ftv.cat, data = lbw,
level = 1,
# No interaction considered
method = "h",
# Exhaustive approach
crit = "aic",
# AIC as criteria
confsetsize = 5,
# Keep 5 best models
plotty = F, report = F, # No plot or interim reports
fitfunction = "lm")
8
summary(glmulti.lm.out@objects[[1]])
end.time<-proc.time()
diff.time1<-end.time-start.time
cat("\n Number of minutes running:", diff.time1[3]/60, "\n \n")
###########################
# GA with no interaction
###########################
start.time<-proc.time()
glmulti.lm.ga.out <glmulti(bwt ~ age + lwt + race.cat + sm + preterm + ht + ui +
ftv.cat, data = lbw,
level = 1,
# No interaction considered
method = "g",
# GA approach
crit = "aic",
# AIC as criteria
confsetsize = 5,
# Keep 5 best models
plotty = F, report = F, # No plot or interim reports
fitfunction = "lm")
summary(glmulti.lm.ga.out@objects[[1]])
end.time<-proc.time()
diff.time2<-end.time-start.time
cat("\n Number of minutes running:", diff.time2[3]/60, "\n \n")
########################################
# Exhuastive case with pairwise interaction
########################################
set.seed(2104)
start.time<-proc.time()
glmulti.lm.out2 <glmulti(bwt ~ -1+age + lwt + race.cat + sm + preterm + ht + ui +
ftv.cat, data = lbw,
level = 2,
# pairwise interaction considered
method = "h",
# Exhaustive approach
crit = "aic",
# AIC as criteria
confsetsize = 5,
# Keep 5 best models
plotty = F, report = F, # No plot or interim reports
fitfunction = "lm")
summary(glmulti.lm.out2@objects[[1]])
end.time<-proc.time()
diff.time3<-end.time-start.time
cat("\n Number of minutes running:", diff.time3[3]/60, "\n \n")
##############################
#GA with pairwise interaction#
##############################
start.time<-proc.time()
set.seed(2104)
glmulti.lm.out.ga2 <glmulti(bwt ~ -1+age + lwt + race.cat + sm + preterm + ht + ui +
ftv.cat, data = lbw,
9
level = 2,
# pairwise interaction considered
method = "g",
# GA approach
crit = "aic",
# AIC as criteria
confsetsize = 5,
# Keep 5 best models
plotty = F, report = F, # No plot or interim reports
fitfunction = "lm")
summary(glmulti.lm.out.ga2@objects[[1]])
ga<-summary(glmulti.lm.out.ga2)
ga$generations
end.time<-proc.time()
diff.time4<-end.time-start.time
cat("\n Number of minutes running:", diff.time4[3]/60, "\n \n")
10