Introduction to Non-Linear Optimization
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
1
Nonlinear Optimization
Note:
Unlike for linear problems, a global optimum for a nonlinear
problem cannot be guaranteed, except for special cases, e.g.,
if you know the space is unimodal, or convex, or monotonicity
exists.
Two standard heuristics that most people use:
1) find local extrema starting from widely varying starting points of the
variables and then pick the most extreme of these extrema (if they
are not the same)
2) perturb a local extremum by taking a finite amplitude step away
from it, and then see whether your routine returns you to a better
point or "always" to the same one.
Question: How would you "automate" a search for a global
extremum?
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
2
Basic Steps in Nonlinear Optimization
•
In its simplest form, a numerical search procedure consists
of four steps when applied to unconstrained minimization
problems:
1) Selection of an initial design in the n-dimensional space, where n
is the number of design variables
2) A procedure for the evaluation of the function (objective function)
at a given point in the design space.
3) Comparison of the current design with all of the preceding
designs.
4) A rational way to select a new design and repeat the process.
•
Constrained minimization requires step for evaluation of
constraints as well. Same applies for evaluating multiple
objective functions.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
3
Nonlinear Optimization Process
• Simplistically, most design tasks seek to find a perturbation to an existing design
which will lead to an improvement. Thus, we seek a new design which is the old
design plus a change:
Xnew = Xold + d X
• Optimization algorithms use the same formula, but apply a two step process:
Xk = Xk-1 + aSk
• You (the engineer) have to provide an initial design X0.
• The optimization will then determine a search direction S k that will improve the
•
design.
Next question is how far we can move in direction Sk before we must find a new
search direction. This is a one-dimensional search since we only have to determine
the value of the scalar a to improve the design as much as possible.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
4
A Good Algorithm
A good algorithm is (among others):
• Robust – algorithm must be reliable for general design applications
and (thus) must theoretically converge to the solution point starting
from any given starting point.
•
•
•
•
General – Should not impose restrictions on the model's constraints
and objective functions.
Accurate – Ability to converge to precise mathematical optimum
point is important, though it may not be required in practice.
Easy to use – by both experienced and inexperienced users. Should
not have problem dependent tuning parameters.
Efficient – To be efficient, the number of repeated analyses should
be kept to a minimum. Hence, an efficient algorithm has 1) a faster
rate of convergence requiring fewer iterations, and 2) least number of
calculations within one (design) iteration.
Note: Tradeoffs have to be made
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
5
Zero and first order algorithms
•
You often must choose between algorithms which need only
evaluations of the objective function or methods that also
require the derivatives of that function.
•
Algorithms using derivatives are generally more powerful, but
do not always compensate for the additional calculations of
derivatives.
•
Note that you may not be able to compute the derivatives.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
6
Basic Descent Methods
•
Basic descent methods are the basic techniques for iteratively
solving unconstrained minimization problems.
•
Important for practical situations because they offer the simplest
and most direct alternatives for obtaining solutions.
•
Also good as a benchmark.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
7
General Basic Descent Method Algorithm
Basic steps:
1. start at an initial point;
2. determine according to a fixed rule a direction of movement; and
3. move in that direction to a (relative) minimum of the objective function
on that line.
4. At the new point, a new direction is determined and the same process
is repeated.
•
The primary difference between algorithms (steepest descent,
Newton's method, etc) is the rule by which successive directions of
movement are selected.
•
The process of determining the minimum point on a line is called line
search.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
8