Engineering Optimization
Concepts and Applications
Fred van Keulen
Matthijs Langelaar
CLA H21.1
A.vanKeulen@tudelft.nl
WB1440 Engineering Optimization – Concepts and Applications
Contents
● Optimization problem checking and simplification
● Model simplification
WB1440 Engineering Optimization – Concepts and Applications
Model simplification
● Basic idea:
Expensive
Cheap
model
Optimizer
● Motivation:
– Replacement of expensive function, evaluated many times
– Interaction between different disciplines
– Estimation of derivatives
– Noise
WB1440 Engineering Optimization – Concepts and Applications
Model simplification (2)
● Drawback: loss of accuracy
● Different ranges: local, mid-range, global
● Synonyms:
– Approximation models
Procedure:
– Metamodels
– Surrogate models
– Compact models
– Reduced order models
WB1440 Engineering Optimization – Concepts and Applications
Extract
information
Construct
approximation
Model simplification (3)
● Information extraction: linked to techniques from
physical experiments: “plan of experiments” / DoE
● Many approaches! Covered here:
– Taylor series expansions
– Exact fitting
– Least squares fitting (response surface techniques)
– Kriging
– Reduced basis methods
– Briefly: neural nets, genetic programming, simplified physical models
● Crucial: purpose, range and level of detail
WB1440 Engineering Optimization – Concepts and Applications
Taylor series expansions
● Approximation based on local information:
1
1
1
f ( x  h)  f ( x )  f ' ( x ) h  f ' ' ( x ) h 2  
1!
2!

1 (n)
  f ( x)h n
Truncation error!
n  0 n!
N
N
1 (n)
1 (n)
n
n
N
  f ( x ) h  o( h )   f ( x ) h
n  0 n!
n  0 n!
● Use of derivative information!
● Valid in neighbourhood of x
WB1440 Engineering Optimization – Concepts and Applications
Taylor approximation example
5
x cos( x / 5)
f 


x / 2
1 e
5
3
Function
Approximation
(x = 20)
th order
th
th
20
rd
43512
order
order
order
st
nd
order
order
x
WB1440 Engineering Optimization – Concepts and Applications
Exact fitting (interpolation)
● # datapoints = # fitting parameters
● Every datapoint reproduced exactly
● Example:
f  a0  a1 x
f2
1 x1  a0   f1 
1 x   a    f 
2  1 

 2
f1
x1
WB1440 Engineering Optimization – Concepts and Applications
x2
Exact fitting (2)
● Easy for intrinsically linear functions:
n
f  a0 f 0  a1 f1  a2 f 2     ai f i
i 1
● Often used: polynomials, generalized polynomials:
f  a  bx1m x2n 
log( f  a)  log b  m log x1  n log x2
● No smoothing / filtering / noise reduction
● Danger of oscillations with high-order polynomials
WB1440 Engineering Optimization – Concepts and Applications
Oscillations
● Referred to as “Runge phenomenon”
5th order
9th order
1
1  25x 2
● In practice: use order 6 or less
WB1440 Engineering Optimization – Concepts and Applications
9th order
polynomial
Least squares fitting
● Less fitting parameters than datapoints
● Smoothing / filtering behaviour
● “Best fit”? Minimize sum of deviations:
squared deviations:
N


2
~
min  | f ( xii )  f ( xii ) |
a
i 1
f
~
f
x
WB1440 Engineering Optimization – Concepts and Applications
Least squares fitting (2)
● Choose fitting function linear in parameters ai :
~
f ( x)  a0 f 0 ( x)  a1 f1 ( x)  a2 f 2 ( x)    am f m ( x)  
~
 f ( x0 )   f 0 ( x0 )
~
 
 f ( x1 )   f 0 ( x1 )
~

  f ( x 2 )    f 0 ( x2 )
    
~
 
 f ( x N )  f 0 ( x N )
● Short notation:
f1 ( x0 )
f 2 ( x0 ) 
f1 ( x1 )
f 2 ( x1 )
f1 ( x2 )
f 2 ( x2 )



f1 ( x N )
f 2 ( xN ) 
~
f  Ma  ε
WB1440 Engineering Optimization – Concepts and Applications
f m ( x0 )   a0    0 
f m ( x1 )   a1   1 
   

f m ( x2 )  a 2     2 
   
  

   
f m ( x N ) am   N 
LS fitting (3)
● Minimize sum of squared errors:



T ~
~
min L  ε ε  f  Ma f  Ma
T
a
(Optimization problem!)


L
T ~
T~
 0  2M f  Ma  2M f  2M T Ma
a
~
M Ma  M f
T
T

WB1440 Engineering Optimization – Concepts and Applications


~
a M M M f
T
1
T
Polynomial LS fitting
● Polynomial of degree m:
~
f ( x)  a0 x0  a1 x1  a2 x 2   am x m  
~
~
2
m
 N  f ( x0
 fi 
x0 xi 2 x0  xx0i aa00   0
) xi 1 
~
2
 ~ 



2
3
mm1

x1 xi x1   xxi1 aa11  
1  f i xi 
1 ) xi 1 
  xi f ( x
2 ~
3
  ~ 2 
4
mm
 2 
2


 
xi f ( x
x 1 
xxi2 aa22  
x2 xi x2
2 f i xi 

2) i


   











     
   



~
m 1
m2 2
2m
m 
m
 x m~
m )xi 1 xmxi xm   xxmi aamm m fi xi 
i f ( x
 
WB1440 Engineering Optimization – Concepts and Applications
Polynomial LS example
1.2
samples
quadratic
6th degree
1
0.8
0.6
0.4
0.2
0
-0.2
-1
-0.8
-0.6
-0.4
-0.2
0
WB1440 Engineering Optimization – Concepts and Applications
0.2
0.4
0.6
0.8
1
Multidimensional LS fitting
● Polynomial in multiple dimensions:
~
f ( x, y )  a0  a1 x  a2 y  
a3 x 2  a4 y 2  a5 xy  
a6 x 3  a7 y 3  a8 x 2 y  a9 xy2    
  ai f i  
● Number of coefficients ai for quadratic polynomial in Rn:
1
m  (n  1)( n  2)
2
Curse of dimensionality!
WB1440 Engineering Optimization – Concepts and Applications
Response surface
● Generate datapoints through sampling:
– Generate design points through Design of Experiments
– Evaluate responses
● Fit analytical model
x3
● Check accuracy
x2
x1
Fractional
2n full factorial
factorialdesign
design
WB1440 Engineering Optimization – Concepts and Applications
Latin Hypercube Sampling (LHS)
● Popular method: LHS
● Based on idea of Latin square:
● Properties:
1
– Space-filling
0.8
– Any number of design points
0.6
– Intended for box-like domains
0.4
– Matlab: lhsdesign
0.2
0
WB1440 Engineering Optimization – Concepts and Applications
0
0.2
0.4
0.6
0.8
1
(LS) Fit quality indicators
● Accuracy? More / fewer terms?

● Examine the residuals
0
– Small
– Random!
xi
● Statistical quality indicators:
– R2 correlation measure:
R2  1
2



 i


~
fi  f
~

 f  f
2
– F-ratio (signal to noise):
F
i
m
   N  m  1
2
i
WB1440 Engineering Optimization – Concepts and Applications
Okay: >0.6
2
Okay: >>1
Nonlinear LS
● Linear LS: intrinsically linear functions (linear in ai):
f ( x)  a0 x 0  a1 x1  a2 x 2    aT p
f ( x)  a0e x  a1 log x  a2 x    aT p
● Nonlinear LS: more complicated functions of ai:
a0 x
f ( x) 
a1 x  a2 x 2  1
● More difficult to fit! (Nonlinear optimization problem)
● Matlab: lsqnonlin
WB1440 Engineering Optimization – Concepts and Applications
LS pitfalls
● Scattered data:
f
x
● Wrong choice of
basis functions:
f
x
WB1440 Engineering Optimization – Concepts and Applications
Kriging
● Named after D.C. Krige, mining engineer, 1951
● Statistical approach: correlation between neighbouring
points
– Interpolation by weighted sum:
N
y ( x)   i ( x, xi ) yi
i 1
– Weights depend on distance
– Certain spatial correlation
function is assumed
(usually Gaussian)
WB1440 Engineering Optimization – Concepts and Applications
Kriging properties
● Kriging interpolation is “most likely” in some sense
(based on assumptions of the method)
● Interpolation: no smoothing / filtering
● Many variations exist!
● Advantage: no need to assume form of
interpolation function
● Fitting process more elaborate than LS procedure
WB1440 Engineering Optimization – Concepts and Applications
Kriging example
● Results depend strongly on statistical assumptions and
method used:
Dataset z(x,y)
WB1440 Engineering Optimization – Concepts and Applications
Kriging interpolation
Reduced order model
● Idea: describing system in reduced basis:
– Example: structural dynamics
  Ku  f 
Mu
~
~
~
Mw  Kw  f
● Select small number of “modes” to build basis
– Example: eigenmodes
WB1440 Engineering Optimization – Concepts and Applications
Reduced order model (2)
● Reduced basis:
u
k  N
ω w
i 1
i
B  ω1 ω 2  ω k 
i

u  Bw
N1
Nk
k1
● Reduced system equations:
  Ku  f
Mu
  KBw  f
MBw


  BT KBw  BT f
BT MBw
WB1440 Engineering Optimization – Concepts and Applications
~
M  B T MB
~
T
K  B KB
 ~
T
f

B
f

u  Bw

kN
NN
kN
N1
Nk
Reduced order models
● Many approaches!
– Selection of type and number of basis vectors
– Dealing with nonlinearity / multiple disciplines
● Active research topic
● No interpolation / fitting, but approximate modeling
WB1440 Engineering Optimization – Concepts and Applications
Example:Aircraft model
Structural model
Mass model
Aerodynamic model
WB1440 Engineering Optimization – Concepts and Applications
Neural nets
WB1440 Engineering Optimization – Concepts and Applications
Neural nets
f(x)
x
output
S(input)
To determine internal neuron parameters,
neural nets must be trained on data.
WB1440 Engineering Optimization – Concepts and Applications
Neural net features
● Versatile, can capture complex behavior
● Filtering, smoothing
● Many variations possible
– Network
– Number of neurons, layers
– Transfer functions
● Many training steps might be required
(nonlinear optimization)
● Matlab: see e.g. nndtoc
WB1440 Engineering Optimization – Concepts and Applications
Genetic programming
● Building mathematical functions using
evolution-like approach
● Approach good fit by crossover and
mutation of expressions
^2
 x1

  x3 
 x2

2
+
/
x1
x3
x2
WB1440 Engineering Optimization – Concepts and Applications
Genetic programming
● LS fitting with population of analytic expressions
● Selection / evolution rules
● Features:
– Can capture very complex
behavior
– Danger of artifacts /
overfitting
– Quite expensive procedure
WB1440 Engineering Optimization – Concepts and Applications
Simplified physical models
● Goal: capture trends from underlying physics through
simpler model:
– Lumped / Analytic / Coarse
● Parameters fitted to “high-fidelity” data
● Refinement: correction function, parameter functions ...
x
Simplified
model
WB1440 Engineering Optimization – Concepts and Applications
Correction
function
f(x)
Model simplification summary
Many different approaches:
– Local: Taylor series (needs derivatives)
– Interpolation (exact fit):
 (Polynomial) fitting
 Kriging
– Fitting: LS
– Approximate modeling: reduced order / simplified models
– Other: genetic programming, neural nets, etc
WB1440 Engineering Optimization – Concepts and Applications
Response surfaces in optimization
●
Popular approach for
computationally expensive problems:
1. DoE, generate samples
Expensive
Cheap
model
(expensive) in part of domain
Optimizer
2. Build response surface (cheap)
3. Perform optimization on response surface (cheap)
4. Update domain of interest, and repeat
●
Additional advantage: smoothens noisy responses
●
Easy to combine with parallel computing
WB1440 Engineering Optimization – Concepts and Applications
Example: Multi-point
Approximation Method
Design domain
Optimum
(Expensive) simulation
Sub-optimal point
Trust region
Response surface
WB1440 Engineering Optimization – Concepts and Applications