Engineering Optimization
Concepts and Applications
Fred van Keulen
Matthijs Langelaar
CLA H21.1
A.vanKeulen@tudelft.nl
Engineering Optimization – Concepts and Applications
Recap / overview
Optimization problem
Negative null form
Special topics
Model
Linear / convex problems
Definition
Sensitivity analysis
Checking
Topology optimization
Solution methods
Unconstrained problems
Constrained problems
Optimality criteria
Optimality criteria
Optimization algorithms
Optimization algorithms
Engineering Optimization – Concepts and Applications
Summary optimality conditions
● Conditions for local minimum of unconstrained problem:
– First Order Necessity Condition:
f  0
– Second Order Sufficiency Condition:
H positive definite
● For convex f in convex feasible domain:
condition for global minimum:
– Sufficiency Condition:
Engineering Optimization – Concepts and Applications
f  0
Stationary point nature summary
y T Hy i
Definiteness H
Nature x*
0
Positive d.
Minimum
0
Positive semi-d.
Valley
0
Indefinite
Saddlepoint
0
Negative semi-d.
Ridge
0
Negative d.
Maximum
Engineering Optimization – Concepts and Applications
Complex eigenvalues?
● Question: what is the nature of a stationary point when
H has complex eigenvalues?
● Answer: this situation never occurs, because H is
symmetric by definition. Symmetric matrices have real
eigenvalues (spectral theory).
Engineering Optimization – Concepts and Applications
Nature of stationary points
● Nature of initial position depends on load (buckling):
F
2
l
k1
F  6, l  2
k1  10, k 2  9.5
k2
dz  l  l cos 1 cos 2
1
1
2
2
  k11  k 2 2  Fdz
2
2
 k11  Fl sin 1 cos  2 
  
0
k2 2  Fl sin  2 cos 1 
0
   
0
Fl sin 1 sin
k1  Fl cos 01 cos 2
k1 2k 2 

  
 Fcrit  min  , 

0 sin k12sin
 Fl
 2
k2  Fl cos 1 lcos l 2
 Fl
 Fcrit  4.75
Engineering Optimization – Concepts and Applications
Nature of stationary points (2)
F 4
3
5
6.75
7

2
1
Engineering Optimization – Concepts and Applications
Unconstrained optimization
algorithms
● Single-variable methods
– 0th order (involving only f )
– 1st order (involving f and f ’ )
– 2nd order (involving f, f ’ and f ” )
● Multiple variable methods
Engineering Optimization – Concepts and Applications
Why optimization algorithms?
● Optimality conditions often cannot be used:
– Function not explicitly known (e.g. simulation)
– Conditions cannot be solved analytically
● Example:
f  x1  x2  x`1  x1e  x2  x2 e  x1
2
2
Stationary points:
 f
2  x1
 x2

1

2
x

e

x
x
0
1
1 2 e
 x
 1
f  0  
 f  1  x x e  x2  2 x e  x1  0
1 2
2
 x2
Engineering Optimization – Concepts and Applications
0th order methods: pro/con
● Strengths:
– No derivatives needed
– Work also for
discontinuous / nondifferentiable functions
– Easy to program
– Robust
Engineering Optimization – Concepts and Applications
● Weaknesses:
– (Usually) less efficient
than higher order
methods (many function
evaluations)
Minimization with one variable
● Why?
– Simplest case: good starting point
– Used in multi-variable methods during line search
● Setting:
min f ( x)
Iterative process:
x
f
s.t. x  x  x
Model
Optimizer
x
Engineering Optimization – Concepts and Applications
Termination criteria
● Stop optimization iterations when:
– Solution is sufficiently accurate (check optimality criteria)
– Progress becomes too slow:
xk  xk 1   x ,
f ( xk )  f ( xk 1 )   f
– Maximum resources have been spent
– The solution diverges
– Cycling occurs
Engineering Optimization – Concepts and Applications
xb
xa
Brute-force approach
● Simple approach: exhaustive search
f
L0
n points:
Final interval size =
Ln
2
Ln 
L0
n 1
x
● Disadvantage: rather inefficient
Engineering Optimization – Concepts and Applications
Basic strategy of 0th order
methods for single-variable case
1. Find interval [a0, b0] that contains the minimum
(bracketing)
2. Iteratively reduce the size of the interval [ak, bk]
(sectioning)
3. Approximate the minimum by the minimum of a simple
interpolation function over the interval [aN, bN]
● Sectioning methods:
– Dichotomous search
– Fibonacci method
– Golden section method
Engineering Optimization – Concepts and Applications
Bracketing the minimum
f
[a0, b0]
x
x1
x2 = x1+
x3 = x2+g
x4 = x3+g2
Starting point x1, stepsize , expansion parameter g: user-defined
Engineering Optimization – Concepts and Applications
Unimodality
● Bracketing and sectioning methods work best for
unimodal functions:
“An unimodal function consists of exactly one
monotonically increasing and decreasing part”
Engineering Optimization – Concepts and Applications
Dichotomous search
● Conceptually simple
idea:
Main Entry: di·chot·o·mous
Pronunciation: dI-'kät-&-m&s also d&Function: adjective
: dividing into two parts
– Try to split interval in half in each step
L0
a0
L0/2
f   f  :
Engineering Optimization – Concepts and Applications
d << L0
b0
Dichotomous search (2)
● Interval size after 1 step (2 evaluations):
1
L1  L0  d 
2
L0
● Interval size after m steps (2m evaluations):
L0
1 

Lm  m  d 1  m 
2
 2 
● Proper choice for
L0
ideal
Lm  m
2
d:

Engineering Optimization – Concepts and Applications
Lm Lideal
L0
m
d


10
10 10  2m
Dichotomous search (3)
● Example: m = 10
ideal
10
L
10
 Lm 
log  
 L0 
m
L0
L0
 10 
2
1024
ideal
L10
L0
d 

10 10240
d
Ideal
interval
reduction
Engineering Optimization – Concepts and Applications
L0
10240
Sectioning - Fibonacci
● Situation:
Fibonacci,
1180?-1250?
minimum
bracketed
between x1
and x3 :
x1
x2
● Test new points and reduce interval
● Optimal point placement?
Engineering Optimization – Concepts and Applications
x4
x3
Optimal
sectioning
● Fibonacci method: optimal sectioning method
● Given:
– Initial interval [a0, b0]
– Predefined total number of evaluations N, or:
– Desired final interval size 
Engineering Optimization – Concepts and Applications
Fibonacci sectioning - basic idea
● Start at final interval and use symmetry and maximum
IN
interval reduction:
IN-1 = 2IN
IN-2 = 3IN
IN-3 = 5IN
IN-4 = 8IN
d << IN
IN-5 = 13IN
I N  j  F j 1 I N
F j  Fibonacci number
I N  j  2  I N  j 1  I N  j
 I k  I k 2  I k 1
Engineering Optimization – Concepts and Applications
Sectioning – Golden Section
● For large N, Fibonacci fraction
b converges to golden
section ratio f (0.618034…):
● Golden section method
uses this constant interval
reduction ratio f
f
1
1
f
Engineering Optimization – Concepts and Applications
FN 1
FN
Sectioning - Golden Section
● Origin of golden section:
I1  I 2  I 3
 I1  fI1  f 2 I1
f  f  1  0  f1, 2
2
I2 = fI1
1 5

2
5 1
f
 0.618034
2
N
● Final interval: I N  f I1
Engineering Optimization – Concepts and Applications
I1
I2 = fI1
I3 = fI2
Comparison sectioning methods
Evaluations
 Lm 
10
log  
 L0 
Ideal dichotomous
interval reduction
Example:
reduction to 2% of
original interval:
N
Golden
section
Fibonacci
Dichotomous
12
Golden section 9
Fibonacci
(Exhaustive
8
99)
● Conclusion: Golden section simple and near-optimal
Engineering Optimization – Concepts and Applications
Quadratic interpolation
● Three points of the bracket define interpolating quadratic
function:
~
f ( x)  ax 2  bx  c
● New point evaluated at
minimum of parabola:
● For minimum: a > 0!
ai
ai+1
bi+1
i
~
b
f '  2ax  b  0  xnew 
2a
● Shift xnew when very close to existing point
Engineering Optimization – Concepts and Applications
xnew
Unconstrained optimization
algorithms
● Single-variable methods
– 0th order (involving only f )
– 1st order (involving f and f ’ )
– 2nd order (involving f, f ’ and f ” )
● Multiple variable methods
Engineering Optimization – Concepts and Applications
Cubic interpolation
● Similar to quadratic interpolation, but with 2 points and
derivative information:
~
f ( x)  ax 3  bx 2  cx  d
ai
Engineering Optimization – Concepts and Applications
bi
Bisection method
● Optimality conditions: minimum at stationary point
 Root finding of
f’
● Similar to sectioning methods, but uses derivative:
f
Engineering Optimization – Concepts and Applications
f’
Secant method
● Also based on root finding of
● Uses linear interpolation
f’
Engineering Optimization – Concepts and Applications
f’
Unconstrained optimization
algorithms
● Single-variable methods
– 0th order (involving only f )
– 1st order (involving f and f ’ )
– 2nd order (involving f, f ’ and f ” )
● Multiple variable methods
Engineering Optimization – Concepts and Applications
Newton’s method
● Again, root finding of
f’
● Basis: Taylor approximation of
f ’:
f '( x  h)  f '( x)  f ''( x)h  o(h2 )
f ' ( x)
h
f " ( x)
● New guess:
f ' ( xk )
xk 1  xk  hk  xk 
f " ( xk )
Engineering Optimization – Concepts and Applications
Linear
approximation
Newton’s method
● Best convergence of all methods:
f’
f’
xk
xk+2
xk
xk+1
● Unless it diverges
Engineering Optimization – Concepts and Applications
xk+1
xk+2
Summary single variable methods
● Bracketing +
 Dichotomous sectioning
 Fibonacci sectioning
 Golden ratio sectioning
0th order
In practice:
additional “tricks”
needed to deal
with:
 Quadratic interpolation
 Cubic interpolation
 Bisection method
1st order
 Multimodality
 Strong
 Secant method
 Newton method
● And many, many more!
Engineering Optimization – Concepts and Applications
2nd order
fluctuations
 Round-off
errors
 Divergence