AMITY INSTITUTE OF SPACE SCIENCE & TECHNOLOGY Classical Optimization Technique Single Variable Optimization M S Prasad 7/15/2015 This lecture note is prepared for students of Graduate engg and is based on open literature and text books. This needs to be read in conjunction with class room discussions. Classical Optimization Techniques :LN -2 The classical optimization techniques are useful in finding the optimum solution or un constrained maxima or minima of continuous and differentiable functions. These are analytical methods and make use of differential calculus in locating the optimum solution. The classical methods have limited scope in practical applications as some of them involve objective functions which are not continuous and/or differentiable. �Yet, the study of these classical techniques of optimization form a basis for developing most of the numerical techniques that have evolved into advanced techniques more suitable to today’s practical problems Single-variable optimization algorithms The algorithms described in this section can be used to solve minimization problems of the following type: Minimize f(x) is the objective function and x is a real variable. The purpose of an optimization algorithm is to find a solution x, for which the function f(x) is minimum. These algorithms are classified into two categories i. Direct methods ii. Gradient based methods Direct methods do not use any derivative in formation of the objective function ; only objective function values are used to guide the search process. However gradient based methods use derivative information (first and/or second order) to guide the search process. Optimality criteria There are three different types of optimal points [ minima] are: Local Optimal x*is said to be a local optimal point, if no point in the neighborhood has a function value smaller than f(x*). Global Optimal point: A point or solution x** is said to be a global optimal point, if no point in the entire search space has a function value smaller than f(x**). Inflection point: x*is an inflection point if f(x*) increases locally as x*increases & decreases locally as x*reduces or f(x*) decreases locally as x* increases and increases locally as x*decreases. Let the objective function f(x) is the chosen search space f'(x)and f''(x) are first and second derivatives. A point x is a minimum if f'(x)=0 & f''(x)>0. If f‘(x)=0,the point is either a minimum, a maximum or an inflection point . Suppose at point x*,the first derivative is zero and the first non- zero higher order derivative is denoted by n, then if n is odd, x*is an inflection point If n is even, x*is a local optimum (i) If the derivative is +ve, x*is a local minimum (ii) If the derivative is –ve,x* is a local maximum Many powerful direct search algorithms have been developed for the unconstrained minimization of general functions. These algorithms require an initial estimate to the optimum point, denoted by xo. With this estimate as starting point, the algorithm generates a sequence of estimates x1,, x2, , by successively searching directly from each point in a direction of descent to determine the next point. The process is terminated if either no further progress is made, or if a point xk is reached (for smooth functions) at which the first necessary condition i.e f ‘(x) = 0 is sufficiently accurately satisfied, in which case x* = xk. It is usually, although not always, required that the function value at the new iterate x i+1 be lower than that at xi An important sub-class of direct search methods, specifically suitable for smooth functions, are the so-called line search descent methods. Basic to these methods is the selection of a descent direction u i+1 at each iterate Xi that ensures descent at xi in the direction ui+1 , i.e. it is required that the directional derivative in the direction ui+1 be negative. General structure of a line search descent method 1. Given starting point x0 and positive tolerances e1, e2 and e3, set i = 1. 2. Select a descent direction u i. 3. Perform a one-dimensional line search in direction u i: i.e. To get minimum 𝛌i 4. Set 5. Test for convergence Else go to step 6 6. set i = i +1 go to step 2 Fig 1.0 Sequence search and steps Finding the Minima A. Bracketing Methods: The minimum of a function is found in two phases. Initially an approximate method is used to find a lower and an upper bound of the minimum. Next, a sophisticated technique is used to search within these two limits to find the optimal solution. A1 Exhaustive Search Method It is the simplest of all search methods. The optimum of a function is bracketed by calculating the function values at a number of equally spaced points. The search begin from a lower bound on the variable and three consecutive function values are compared at a time based on the assumption of uni modality of the function. Based on the outcome of comparison , the search is either terminated or continued by replacing one of the three points with a new point. Algorithm Step 1 Say n is the number of intermediate points then set Step II If Step III the minimum point lies between ( x1 ,x3 ) , else Is If yes , go to Step2. Else no minimum exists in (a, b) or a boundary point (a or b) is the minimum point . A2. Bounding Phase Method Step 1 Choose an initial guess xo and an increment Δ Set counter k=0 Step 2 If f( xo - |Δx|) >= f(xo) >= f( xo + |Δx|) then Δ is positive f( xo - |Δx|) <= f(xo) <= f( xo + |Δx|) then Δ is negative else go to step 1 Step 3 Set xk+1 = xk + 2kΔ Step4 If f (xk+1 ) < f (xk ) then Set k=k+1 and go to Step 3 else, the minimum lies in the interval ( x k-1 , x k+1 ) and terminate . A3 Region elimination methods: Once the minimum point is bracketed, a more sophisticated algorithm is used to improve the accuracy of the solution. Region elimination methods are used for this purpose. The fundamental rule for region elimination method is as follows : - Fig 2.0 Unimodal Function Xj Consider a unimodal function as above . The two points x1 and x2 lie in the interval (a,b) and satisfy x1< x2. For minimization, the following conditions apply If f (x 1) > f(x2) then the minimum does not lie in( a,x1) If f (x1) < f(x2) then the minimum does not lie in (x2,b) If f (x1) = f(x2) then the minimum does not lie in (a,x1) and (x2,b) A little variation to this algorithm is based on elimination of portion of search i.e. known as Interval Halving Method . In the figure above we have an intermediate point as Xj . The possibilities of functional search now could be as under : a. If f(x1 ) < f(xj) then minimum cannot lie beyond Xj . Hence we can reduce the search space between X1 or a and xj , thus reducing it by half. b. If f (x 1) > f(xj) then minimum cannot lie between a and X1 . thus reduction of ¼ search space. This process can be continued till the small search space is not achieved . Algorithm Step I Choose lower and Upper bound a and b : a small value Є for accuracy . Xj = a+b/2 ; Lo = L = b – a ; compute f(xj) . Step II Set X1 = a+ L/4 ; x2 = b- L/4 ; compute f(X1 ) and f(X2 ) Step III If f(x1 ) < f(xj) then b = xj , xj = x1 go to step V else step IV if f(x2 ) < f(xj) then a = xj , xj = x2 go to step V else a = x1 , b= x2 go to step V. Step v Set L = b-a ; if |L| < Є terminate . else Step II. A4 Fibonacci search method { Golden ratio } The search interval is reduced according to Fibonacci numbers. Fn = Fn-1 + Fn-2 where n = 2,3,4 and F0 = F1 = 1 Fig 2.1 Fibonacci search points(x1and x2) Algorithm: Choose a lower bound a and upper bound b . Set L = b-a. Assume desired iteration to be n . Set K = 2. Step 2 Compute Set and Step 3 . Compute one of f ( X1) or f(X2) which was not evaluated earlier. Use the fundamental region elimination rule to eliminate a region. Set new a and b. Step 4 Is k=n? If no, set k=k+1 and go to Step2. Else terminate . Golden Section search C. Gradient based Method Despite finding the derivative is difficult this method is used commonly due to simple calculations involved Newton Raphson Method : Considering Taylor series expansion Algorithm: 1.Choose initial guess X 1 and small value Є Set k = 1. Compute f ‘ (X1 ). 2.Compute f “ (XK) 3.Calculate Xk+1 = Xk - ( f ’ ( X k) / f “ ( Xk ) ) Compute f “ ( xk ) 4. If | f ‘ ( xk+1 ) | less than Є terminate . Else k = k+1 and go to Step 2. The Convergence depend on initial guess. _____________________________________________________________________________ . Reference 1. Arora JS. Introduction to Optimum Design (2nd edn). Elsevier Academic Press, 2004. 2. Barbosa HJC and Lemonge ACC. A new adaptive penalty scheme for genetic algorithms. Information Sciences 2003; 156(3–4):215–251. 3. Cox SE, Haftka RT, Baker CA, Grossman B, Mason WH and Watson LT. A comparison of global optimization methods for the design of a high-speed civil transport. Journal of Global Optimization 2001; 21(4):415–432. 4. Dorigo M, Maniezzo V and Colorni A. Ant system: optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man and Cybernetics, Part B 1996; 26(1):29–41.