advertisement

Advanced Topics in Image Analysis and Machine Learning Image Degradation and Restoration Week 02 Faculty of Information Science and Engineering Ritsumeikan University Today’s class outline – Overview of Soft Computing Definition Applications Techniques – Genetic Algorithms Introduction to Genetic Algorithms – Image Restoration Project Introduction Soft Computing: A Brief Introduction Soft vs. Hard Computing Hard Computing, the normal programming that we do, requires precise and known in advance algorithms that generate the exact solution Soft Computing differs from conventional hard computing in that, unlike hard computing, it is tolerant of imprecision, uncertainty, partial truth, and approximation Soft Computing is to find approximate solutions to both imprecisely and precisely formulated problems Soft Computing: The Idea Exploit the tolerance for imprecision, uncertainty and partial truth to achieve tractability, robustness, and low solution cost There are many intractable problems in the world that have no known polynomial algorithm available for their solutions (e.g. VLSI layout design, cargo placement / packing optimisation, etc.) Soft Computing Applications Handwriting and speech recognition Image processing and data compression Automotive system control and manufacturing Decision-support systems (planning and scheduling, etc.) Artificial vision systems Consumer electronics control … Techniques of Soft Computing Soft Computing includes three major complementary groups of techniques: - Evolutionary Computation: Genetic Algorithm (GA) and Genetic Programming - Neurocomputing: Artificial Neural Networks (ANN) and Machine Learning - Fuzzy Sets and Fuzzy Logic, Rough Sets, and Probabilistic Reasoning Overview of the Techniques Evolu&onary Computa&on: the algorithms for search and op&miza&on Neurocompu&ng: the algorithms for learning and curve ﬁ<ng Fuzzy Logic and Probabilis&c Reasoning: the algorithms for dealing with decision‐making imprecision and uncertainty, and Rough Sets: the approach to handle uncertainty arising from the granularity in the domain of discourse Evolutionary Computation: Genetic Algorithm The idea (by John Holland, in 1970s): - Evolution works biologically so, maybe, it will work with simulated environments - Each possible solution (whether good or bad) is represented by a chromosome – a pattern of bits which works as DNA. Then: Determine the better solutions Mate these solutions to produce new solutions which are (hopefully) occasionally better than the parents Repeat this for many generations Artificial Neural Network (ANN) ANN Training ANN’s are trained in a variety of ways. During the training process, the weights for links among the diﬀerent neurons are modiﬁed un&l the desired output is generated for each input case used in the training process OUTPUT .9 .2 .2 .4 1.1 1.7 .2 .3 3 .4 .2 .1 2 INPUT 1 Fuzzy Sets and Fuzzy Logic Sets with fuzzy boundaries - Consider a set of tall people Crisp set A 1.0 Fuzzy set B 1.0 .9 Membership .5 function 172cm Heights 172 186 Heights Rough Sets A rough set is a set with vague boundaries. By comparison, a regular set has sharp boundaries, that is, elements on the boundary either belong to the set or do not belong to it. Rough Sets Set membership of elements on the boundary of a rough set is undetermined, i.e., it is not known whether they belong to the set or not. A rough set is characterised by its lower approximation, that is, a collection of elements which belong to the set for sure, and its upper approximation, that is a collection of elements which may belong to a set. A boundary of a rough set is defined as a difference between the upper approximation and the lower approximation. Simulated Annealing: The Problem A simple, smooth 2D problem Global Maximum A real-world problem Optimisation problem: Search of the global maximum Simulated Annealing: The Idea (Physical) annealing is the process of slowly cooling down a substance (like a heated liquid metal) Simulated annealing is a stochastic optimisation method implemented by the Metropolis Algorithm (Monte Carlo): – Current thermodynamic state = solution (hopefully, the global maximum) - Energy equation = objective function - “Ground” (stable) state = global optimum Genetic Algorithm (GA) OVERVIEW A class of probabilistic optimisation algorithms Inspired by the biological evolution process Uses concepts of “Natural Selection” and “Genetic Inheritance” (Darwin 1859) Originally developed by John Holland (1975) Special Features: – Traditionally emphasizes combining information from good parents (crossover) – There are many GA variants, e.g., reproduction models, operators GA overview (cont) Particularly well suited for hard problems where little is known about the underlying search space Widely-used in business, science and engineering Holland’s original GA is now known as the simple genetic algorithm (SGA). Other GAs use different: – Representations – Mutations – Crossovers – Selection mechanisms Function Optimisation GA's are useful for solving multidimensional problems containing many local maxima (or minima) in the A real-world problem solution space A simple optimisation problem (no need to use a GA to solve this!) global local A standard method of finding maxima or minima is via the gradient decent (gradient ascent) method global local I found the top! Problem: this method may find only a local maxima! Genetic Algorithm: the Idea My height is 13.2m My height is 10.5m My height is 7.5m My height is 3.6m The Genetic Algorithm uses multiple climbers in parallel to find the global optimum Genetic algorithm – some iterations later A climber has approached the “global maximum” I found the top! GA Stochastic operators Selection replicates the most successful solutions found in a population at a rate proportional to their relative quality Crossover takes two distinct solutions and then randomly mixes their parts to form novel solutions Mutation randomly perturbs a candidate solution The Evolutionary Cycle fittest parents selection modification modified offspring initiate & evaluate population parents evaluated 䇺strong䇻 offspring evaluation deleted members discard GA Example: the MAXONE problem Suppose we want to maximise the number of ones in a string of 10 binary digits A gene can be encoded as a string of 10 binary digits, e.g., 0010110101 The fitness f of a candidate solution to the MAXONE problem is the number of ones in its genetic code, e.g. f(0010110101) = 6 We start with a population of n random strings. Suppose that n = 6 Example (initialisation) Our initial population of parent genes is made using random binary data: s1 = 1111010101 f (s1) = 7 s2 = 0111000101 f (s2) = 5 s3 = 1110110101 f (s3) = 7 s4 = 0100010011f (s4) = 4 s5 = 1110111101 f (s5) = 8 s6 = 0100110000f (s6) = 3 The fitness f of a parent gene is simply the sum of the bits. Selection Selection is an operation that is used to choose the best parent genes from the current population for breeding a new child population Purpose: to focus the search in promising regions of the solution space Example (Selection) Next we apply fitness proportionate selection with the roulette wheel method: f (i ) Individual i will have a ∑ f (i ) i probability to be chosen We repeat the extraction as many times as the number of individuals we need to have the same parent population size (6 in our case) 1 n Area is Proportional to fitness value 2 3 4 Example (selection continued) Suppose that, after performing selection, we get the following population: Selected parents s` s1` = 1111010101 (s1) s2` = 1110110101 (s3) s3` = 1110111101 (s5) s4` = 0111000101 (s2) s5` = 0100010011 (s4) s6` = 1110111101 (s5) Original parents (s) Example (crossover) • Next we mate parent strings using crossover. • For each pair of parents we decide according to a crossover probability (for instance 0.6) whether to actually perform crossover or not. • Suppose that we decide to actually perform crossover only for pairs (s1`, s2`) and (s5`, s6`). • For each pair, we randomly choose a crossover point, for instance bit 2 for the first and bit 5 for the second parent Example (crossover cont.) Before crossover: s1` = 1111010101 s2` = 1110110101 s5` = 0100010011 s6` = 1110111101 After crossover: s1`` = 1110110101 s2`` = 1111010101 s5`` = 0100011101 s6`` = 1110110011 Note: sometimes crossover results in no changes to the pair! Example (mutation) The final step is to apply random mutation: for each bit in the current gene population we allow a small probability of mutation (for instance 0.05) Before applying mutation: After applying mutation: s1`` = 1110110101 s2`` = 1111010101 s3`` = 1110111101 s4`` = 0111000101 s5`` = 0100011101 s6`` = 1110110011 s1``` = 1110100101 s2``` = 1111110100 s3``` = 1110101111 s4``` = 0111000101 s5``` = 0100011101 s6``` = 1110110001 Fitness: f (s1``` ) = 6 f (s2``` ) = 7 f (s3``` ) = 8 f (s4``` ) = 5 f (s5``` ) = 5 f (s6``` ) = 6 Purpose: mutation adds new information that may be missing from the current population Example: Results • In one generation, the total population fitness changed from 34 to 37, thus improved by ~9% • At this point, we go through the same process all over again (repetition), until a stopping criterion is met Another example – Maximise X2 Simple problem: maximise y=x2 over the x interval {0,1,…,31} GA approach: • Representation: binary code, e.g. 01101 ↔ 13(10 • Population size: 4 genes (parents) • Random initialisation • Roulette wheel selection • 1-point crossover, bit-wise mutation We will show one generational cycle as an example 2 x example: selection • Make sure you understand this slide! You will implement something similar during your image restoration coding project! Probi calculation for gene S1: Prob(169) = 169/1170 = 0.144 Expected count(S1 ) = Probi * n = 0.14 * 4 = 0.58 x2 example: crossover • Each pair of genes may undergo crossover. • The crossover points are randomly selected. • Notice that, after crossover, the average population fitness increased from 293 to 439, and the best genes fitness increased from 576 to 729! x2 example: mutation • All gene bits may undergo mutation (based on the mutation rate). • Notice that, after mutation, the average population fitness increased from 439 to 588(the best genes fitness did not change though)! GA Group Projects • Today we will form teams of several students; • Each team will implement a GA in Matlab (or C/Java/VB?) to restore a corrupted image: • Each team should have one good programmer, and access to a notebook computer (preferably with Matlab)! • You will submit a written report and give a short presentation in week 15 GA Group Project: details The form of the corruption source is additive noise: N(row,col)= NoiseAmp×sin([2π×NoiseFreqRow×row]+[2π ×NoiseFreqCol×col])) Teams must code a simple GA that optimises the three unknown constants NoiseAmp, NoiseFreqRow, and NoiseFreqCol such that the restoration error (the difference between the original and GA-optimised restored image) is minimised. To make things easy, we will measure the average per-pixel restoration error, thus: Restoration error = (Ioriginal + NoiseGA)-Icorrupted where Ioriginal is the original uncorrupted Lena image, Icorrupted is the corrupted image (I will give you), and NoiseGA is the modelled GA corruption noise using the noise equation above. GA Group Project: details Each iteration of your GA will, for each gene in the population: – Generate new values for NoiseAmp, NoiseFreqRow, and NoiseFreqCol. – Corrupt the original image using the equation N(row,col)=NoiseAmp×sin([2π×NoiseFreqRow×row]+[2π×NoiseFreqCol×col])) – Measure the restoration error (subtract the corrupted image from the original image). This becomes the (inverse of) this gene’s fitness – Make new child genes using selection, crossover, and mutation functions. The search ranges for the three variables are: – NoiseAmp 0 to 30.0 – NoiseFreqRow 0 to 0.01 – NoiseFreqCol 0 to 0.01 Each gene encodes all three variables. If you use 1 byte per variable, each gene will be 24-bits, if you use 2-bytes per variable, 48 bits: 10110111 01010001 11001010 (24-bits per gene) NoiseAmp NoiseFreqRow NoiseFreqCol You need to map the (binary) integer values of each gene to floating point values for the variables. I.e, for NoiseAmp, 00000000=0.0 and 11111111=30.0 Next Lecture We will learn more about Genetic Algorithms (GAs) We will discuss the image restoration project. Read: Gonzalez and Woods Access to the course website: http://www.ritsumei.ac.jp/~gulliver/iaml Homework: Project Preparation Start coding your GA. User inputs are population size (integer, e.g., 50), crossover rate (%, integer, e.g. 60), mutation rate (%, integer, e.g. 5), and total iterations (integer, e.g. 100). Make arrays to hold the gene binary values Fill the arrays with random binary data Map the gene’s binary values to the three noise parameters’ values (floating point) Using the equation N(row,col)=NoiseAmp*sin([2π*NoiseFreqRo w*row]+[2π*NoiseFreqCol*col])) calculate the corruption noise for each pixel of the image. Remember, the noise values can be negative, so use signed data types.