28th, June, 2010,
28th AIAA Applied Aerodynamics Conference
Design Optimization Utilizing
Gradient/Hessian Enhanced Surrogate Model
Wataru YAMAZAKI,
Markus P. RUMPFKEIL,
Dimitri J. MAVRIPLIS
Dept. of Mechanical Engineering,
University of Wyoming, USA
Outline
*Background
- Efficient CFD Gradient/Hessian calculations
- Surrogate Model Enhanced by Gradient/Hessian
- Uncertainty Analysis
*Objectives
*Surrogate Model Approaches
- Kriging
- Direct and Indirect Gradient-enhanced Kriging
- Gradient/Hessian-enhanced Kriging Approaches
*Results & Discussion
- Analytical Function Fitting
- Aerodynamic Data Modeling
- 2D Airfoil Drag Minimization
- Uncertainty Analysis at Optimal Airfoil
*Conclusions
-2-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Background~ Efficient CFD Hessian Calculation
An efficient CFD Hessian calculation method
by Adjoint method and Automatic Differentiation (AD)
For steady flow
i.
Solutions for grid deformation / flow residual equations
sD, xD  0
ii.
RD, xD, wD  0
Adjoint solutions for flow / grid deformation equations
T
T
 R 
 F 

  
 0

w

w




T
T
 s 
 F
T R 

0

 

x 
 x 
 x
iii.
Ndv linear solutions each for dx/dDj and dw/dDj
iv.
Ndv(Ndv+1)/2 cheap evaluations for each Hessian component
2
d F
dD j dDk
  jk F     jk R     jk s 
T
T
M.P. Rumpfkeil and D.J. Mavriplis, AIAA-2010-1268
“Efficient Hessian Calculations using Automatic Differentiation and the Adjoint Method”
-3-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Background~ Efficient CFD Hessian Calculation
An efficient CFD Hessian calculation method
by Adjoint method and Automatic Differentiation (AD)
Grid Deformation
Flow Residual
Flow Adjoint
Mesh Adjoint
dx/dD1
dw/dD1
dx/dD2
dw/dD2
......
dx/dDNdv
dw/dDNdv
Gradient and Hessian
M.P. Rumpfkeil and D.J. Mavriplis, AIAA-2010-1268
“Efficient Hessian Calculations using Automatic Differentiation and the Adjoint Method”
-4-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Background~ Approximate CFD Hessian
For steady flow, a special form of objective function
F
K
w F

2
1
k
k

target 2
k
F
k 1
e.g. F 
 dFk
  wk 
dDi dD j
k 1
 dDi
2
d F
K
 dFk
  wk 
k 1
 dDi
K

w C
2
1
L
L
C

target 2
L

 wD C D  C

target 2
D
T
2
K
 dFk
d
Fk
target


 dD   wk Fk  Fk
dDi dD j
k 1
j

T
 dFk

 dD
j

0
 Last approximation is accurate only nearly optimum
 Approximate Hessian only requires the first-order derivatives
M.P. Rumpfkeil and D.J. Mavriplis, AIAA-2010-1268
“Efficient Hessian Calculations using Automatic Differentiation and the Adjoint Method”
-5-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Background~ Uncertainty Analysis
 Uncertainty due to manufacturing tolerances
in-service wear-and-tear etc
 Analysis of mean/variance/PDF of
objective function w.r.t. fluctuation of design variables
Full Monte-Carlo Simulation
 Thousands/Millions exact function calls
 Accurate and easy, but computationally expensive
Moment Method
 Taylor series expansion by grad/Hessian at the center
 No information about PDF
Inexpensive Monte-Carlo Simulation
 Thousands/Millions surrogate model function calls
 Much lower computational cost
-6-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Objectives
The efficient adjoint gradient/Hessian calculation methods
will be effective…
 for more efficient global design optimization
with G/H-enhanced surrogate model approach
 for more accurate and cheaper uncertainty analysis
by inexpensive Monte-Carlo simulation
with G/H-enhanced surrogate model
Development of gradient/Hessian-enhanced surrogate models
Application to design optimization and uncertainty analysis
-7-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Kriging, Gradient-enhanced Kriging
Kriging model approach
- originally in geological statistics
Two gradient-enhanced Kriging (cokriging or GEK)
 Direct Cokriging
Gradient information is included in the formulation
(correlation between func-grad and grad-grad)
 Indirect Cokriging
Same formulation as original Kriging
Additional samples are created by using the gradient info
Kriging model by both real and additional pts
2D example
: Real Sample Point
: Additional Sample Point
-8-
x add  x i  x
yadd  yxi   x
T
yxi 
x
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Gradient/Hessian-enhanced Kriging
Indirect Approach
x add  x i  x
yadd  yxi   x G 
T
1
x Hx
T
2
2D example
: Real Sample Point
: Additional Sample Point
Arrangements to Use Full Hessian / Diagonal Terms
Major parameters : distance between real / additional pts
number of additional pts per real pt
Worse matrix conditioning with smaller distance
larger number of additional pts
Severe tradeoffs for these parameters
-9-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Gradient/Hessian-enhanced Kriging
Direct Approach
Consider a random process model estimating a function value
by a linear combination of function/gradient/Hessian components
n
n
yˆ  x  

i 1
wi y i 

i 1
n 
i yi   i yi
'
''
i 1
Minimizing Mean-Squared-Error (MSE) between exact/estimated function
 n
MSE  yˆ  x    E    w i y i 
  i 1
with an unbiasedness constraint
n

i 1
n 

i y   i y  Y 
i 1

'
i
''
i
2



n
w
i
1
i 1
Solving by using the Lagrange multiplier approach
J
J
J
J
 n

J  MSE  yˆ  x       w i  1  



0
wi
i
i

 i 1

-10-
 for

i

Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Gradient/Hessian-enhanced Kriging
Direct Approach
Introducing correlation function for covariance terms
Correlation is estimated by distance between two pts with radial basis function

Cov Z 
xi

    R x , x 
Z  x j    R x i , x j 
2
Cov Z  x i  ,
 ,  Z x j 
x
k
2
i
j
x j
k


Unknown parameters are determined by the following system of equations
R
 T
F
F   x  r 
 ~   
0     1 
x
T
  w1 ,  , 1 ,  , 1 ,  
Final form of the gradient/Hessian-enhanced direct Kriging approach is
T
1
yˆ  x     r x  R  Y   F 
  F R F 
T
1
1
F
T

1
R Y
r x 
R
mean constant
term
correlatio ns between x and observed

Y  y1 ,  , y1 ,  , y1 , 
'
''

T
correlatio ns between observed
data
data
exact informatio n at given samples
-11-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Gradient/Hessian-enhanced Kriging
Direct Approach

 R  x1 ,x1 




 R  x n ,x1 

  R x ,x 
1
1

1
 x1

R  

  R  x n , x 1 

N dv
 x n
2
  R  x1 ,x1 

2 1
  x1


2
  R  x n  , x 1 
 2 Ndv
  x n 

R  x1 ,x n 



R x n ,x n 
T
1
yˆ  x     r x  R  Y   F 
 R  x1 ,x1 

 x1

 R  x1 ,x n 
1


 R  x n , x 1 
 R  x1 ,x1 
 x n  dv

 R  x n , x n 
 x1

2
 R  x1 ,x n 
N
1
2
 R  x n  , x 1 
2
2



 x n 

2
 R  x n  , x n 
Ndv
2










 n  n   n    n  n   n  











 x
Correlations
between
F-F,
F-G,
G-G,
F-H,
G-H
and H-H
 R
 R
R
 R
 R
 function

Up 
to 4th xorderderivatives
 x x
x
 x x
 x  xof correlation



by TAPENADE


 Differentiation

Automatic
 R
 R
 R
 R
R



No sensitive
parameter
 x x
 x x
 x
x x
x
 R
 R thanindirect
R
matrix
R
R
conditioning
Better
approach



 x1 ,x n 
 x1
1
2
 x n , x n 
2
2
 x1 ,x n 
 x

2
 R  x n  , x n 
2


1
1
 x n 
2
Ndv
N dv
n
3
1
1
2
1
1
1
1
 x n d v  x1
N
1
N dv
n

4
 x 1 , x n 
 x n  d v  x n  d v
 x1  x n d v
2
N
-12-
N dv
n
N
2
2
1
1
2
1
2
N
N dv
n
N dv
n 
4
 x
1
1
1
1
 x n  , x n  
2
 x1 ,x1 
 x  x

4
 R  x 1 , x n  
1
1
3
N dv
n
1
1
 x n  , x 1 
N dv
n 
2
 x 1 , x n 
 x x

3
 R  x n  , x n  
2

3
2
3
1
1
1
1
Ndv
n 
2
 x1 ,x1 
2
 x n , x n 
3
 x1 ,x1 
3
N dv
n
2
1
1
1
2
 x 1 , x n 
1
1
 x x

3
 R  x n  , x 1 
2
2
2
 x n , x 1 
N dv
n
2
N
 x1 ,x1 
1
1
 x1
 x n  dv
2
 x n  , x 1 
N dv
n 
 x1
2
1


 R  x n  , x n  
4

 x n  d v  x n  d v
2
N
2
N
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Infill Sampling Criteria for Optimization
 How to find promising location on surrogate model ?
 Maximization of Expected Improvement (EI) value
 Potential of being smaller than current minimum (optimal)
 Consider both estimated function and uncertainty (RMSE)
 y min  ˆy  x  
 y min  ˆy  x  
ˆ



 y x 
  s 

s
s




Exact Function
  EI

 0,
ˆ

y

Sample Points
Kriging

 0 
s

 EI
RMSE
EI
1.0
1.0E-02
0.8
8.0E-03
0.6
6.0E-03
0.4
4.0E-03
0.2
2.0E-03
0.0
0.0E+00
EI
Function / RMSE
EI  x    y min
0.0
0.2
-13-
0.4
0.6
Design Variable
0.8
1.0
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Results & Discussion
2D Rastrigin Function Fitting
y  x   20  x1  x 2  10 cos  2  x1   cos  2  x 2 
2
2
80 samples by Latin Hypercube Sampling
Direct Kriging approach
Exact Rastrigin Function
Gradient/Hessian-enhanced
Function-based
Gradient-enhanced
Kriging
-15-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
5D Rosenbrock Function Fitting
F:
FG:
FGHd:
FGH:
Function-based Kriging
Gradient-enhanced
G/diag. Hess-enhanced
G/full Hess-enhanced
RMSE .vs. Number of sample points
Superiority in direct Kriging approaches
thanks to exact enforcement of derivative information
better conditioning of correlation matrix
-16-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Validation on Rosenbrock Func.
 1  x 
N d v 1
y x  
i
2


2 2
 100 x i  1  x i

i 1
CDFs of Full-MC and IMC
1.0
1.E+03
0.9
1.E+02
0.8
F
1.E+01
CDF
Objective Function
Optimization History
1.E+04
Direct_FG
1.E+00
Direct_FGH
Indirect_FGH
1.E-01
Full-MC
0.7
IMC_F
0.6
IMC_FG
IMC_FGH
0.5
1.E-02
0.4
1.E-03
0.3
0
100
200
300
0
Number of Sample Points
Minimization of 20D Rosenbrock
30 initial sample points by LHS
EI-based infill sampling criteria
Faster convergence in
G/H-enhanced direct approach
10
20
30
40
50
60
70
80
90
Function Value
Uncertainty analysis on 2D Rosenbrock
5 sample points for surrogate model
(No sample point on the center location)
Superior performance in
G/H-enhanced Inexpensive MC (IMC)
-17-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Aerodynamic Data Modeling
Unstructured mesh CFD
Steady inviscid flow, NACA0012
20,000 triangle elements
Mach Number
[0.5, 1.5]
Angle of Attack[deg] [0.0, 5.0]
21x21=441 validation data
Exact Hypersurface of Lift Coefficient
-18-
Exact Hypersurface of Drag Coefficient
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Aerodynamic Data Modeling
Exact
Function-based
Gradient-enhanced
Kriging
Cl
Cd
Adjoint gradient is helpful to construct accurate surrogate model
CFD Hessian is not helpful due to noisy design space
-19-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
2D Airfoil Shape Optimization
Unstructured mesh CFD
Steady inviscid flow, M=0.755
NACA0012, 16 DVs for Hicks-Henne function
Objective function of inverse design form
F

w C  C
2
1
l
l

target 2
l

 wd Cd  C

1.0
target 2
d
0.9
0.8
0.7

1
2
Cl  0.675
2

100
2
Cd  0.000
2
H(x)
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Exact / Approximate CFD Hessian available
Computational time of F :
2 min,
FG :
4 min,
FGHapprox. : 6 min,
FGHexact : 36 min (4 min in parallel)
Geometrical constraint for sectional area
0.0
-20-
0.1
0.2
0.3
0.4
0.5
x
0.6
0.7
0.8
0.9
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
1.0
2D Airfoil Shape Optimization
Start from 16 initial sample points which only have function info
Gradient/Hessian evaluations only for new optimal designs
Faster convergence in derivative-enhanced surrogate model
Best design in gradient/exact Hessian-enhanced model
-21-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
2D Airfoil Shape Optimization
NACA0012 (Baseline)
Optimal by G/exact H-enhanced model
Towards supercritical airfoils
Shock reduction on upper surface
-22-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
2D Airfoil Shape Uncertainty Analysis
Geometrical uncertainty analysis at optimal airfoil
Center = optimal obtained by Grad/exact H model
optimal (center) ±0.1 airfoil
Comparison between
2nd order Moment Method (MM2)
using gradient/Hessian at the center
Inexpensive Monte-Carlo (IMC1)
using final surrogate model obtained in optimization
Inexpensive Monte-Carlo (IMC2)
using different G/H-enhanced model by 11 samples
Full Non-Linear Monte-Carlo (NLMC)
using 3,000 CFD function calls
-23-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
2D Airfoil Shape Uncertainty Analysis
MM2
using derivative at the center
IMC1
using G/H surrogate model
obtained in optimization
IMC2
using different G/H model
by 11 samples (for st.devi.=0.01)
NLMC
using 3,000 CFD function calls
Mean of objective w.r.t. standard deviation of all design variables
IMC showed good agreement with NLMC at smaller st. devi.
Necessity of additional sampling criteria for total model accuracy ?
Promising IMC with much cheaper computational cost
-24-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Concluding Remarks / Future Works
Development of gradient/Hessian-enhanced Kriging models
Application to design optimization and uncertainty analysis
Direct Kriging approach is superior to indirect approach
More accurate fitting on exact function
Faster convergence towards global optimal design
Promising inexpensive Monte-Carlo simulation at much lower cost
Application to higher-dimensional / complicated design problem
Robust design with inexpensive Monte-Carlo simulation
Gradient/Hessian vector product-enhanced approach
Thank you for your attention !!
-25-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Appendix
Moment Method
Taylor series expansion by grad/Hessian at the center
No information about PDF
1st order Moment Method
 MM 1  F  x c 
 MM 1 
2
N dv

i 1
 dF

 dD i

D
i
xc




2
2nd order Moment Method
 MM
2
  MM 1 
1
N dv

2
i 1

2
MM 2

2
MM 1

1





2
Di
xc

2
 d F
  dD dD
j 1
i

N dv N dv

2
i 1
-27-
 d 2F

 dD i2

DD
i
j
xc
j




2
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Gradient/Hessian-enhanced Kriging
Implementation Details
Correlation function of a RBF
1
6
2 2
 1   h  35  h  18  h  3
scf  , h    3
0



for h  1 
else
Estimation of hyper parameters
by maximizing likelihood function with GA
Correlation matrix inversion by Cholesky decomposition
Search of new sample point location
by maximizing Expected Improvement (EI) value with GA
-28-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Infill Sampling Criteria for Optimization
 How to find promising location on surrogate model ?
 Expected Improvement (EI) value
 Potential of being smaller than current minimum (optimal)
 Consider both estimated function and uncertainty (RMSE)
 y min  ˆy  x  
 y min  ˆy  x  
ˆ



 y x 
  s 

s
s




Exact Function
  EI

 0,
ˆ

y

Sample Points
Kriging

 0 
s

 EI
RMSE
EI
1.0
1.0E-02
0.8
8.0E-03
EI-based criteria have good balance
between global/local searching
0.6
6.0E-03
EI
Function / RMSE
EI  x    y min
0.4
4.0E-03
0.2
2.0E-03
0.0
0.0E+00
0.0
0.2
-29-
0.4
0.6
Design Variable
0.8
1.0
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
5D Rosenbrock Function Fitting
# of pieces of information = sum of # of F/G/H net components
To scatter samples is better than concentration at limited samples
Approximated computational time factor
N sample
TF 
T ,
i
Ti  1 / 2 / 3,
if i has
F / FG / FGH
i 1
G/H-enhanced surrogate model provides better performance
with efficient Gradient/Hessian calculation methods
-30-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
1D Step Function Fitting
1.5
Function Value
1.0
Exact
Samples
0.5
F
FG
FGH
0.0
-0.5
0.0
0.2
0.4
0.6
0.8
1.0
Design Variable
Much better fit by G/H-enhanced direct Kriging
-31-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Minimization of 20D Rosenbrock Func.
1.E+04
Objective Function
1.E+03
F
FG
FGH
1.E+02
1.E+01
1.E+00
1.E-01
1.E-02
0
3000
6000
9000
12000
15000
Computational Time [sec]
Minimization of 20 dimensional Rosenbrock function
No computational cost for Func/Grad/Hess evaluation
Expensive for construction - likelihood function maximization
- inversion of correlation matrix
Parallel computation for the likelihood maximization problem
-32-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Uncertainty Analysis
150
1.0
130
0.9
Full-MC
110
90
MM2
70
IMC_F
50
IMC_FG
0.6
IMC_FGH
0.5
30
CDF at St. Devi.=0.15
0.8
CDF
Mean of Function
Uncertainty analysis at (1.0,1.0) on 2D Rosenbrock
5 sample points for surrogate model approaches
(No sample point on the center location)
2nd order Moment Method (MM2) by G/H at the center
Superior results in G/H-enhanced Inexpensive MC (IMC)
0.7
0.4
10
-10
0.3
0.0
0.1
0.2
0.3
0.4
0.5
0
Standard Deviation of DVs
10
20
30
40
50
60
70
80
90
Function Value
St. Devi. = 0.15 means the possibility within -0.15<dx<0.15 is about 70%
-33-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
Aerodynamic Data Modeling
 NACA0012
 M=1.4
 AoA=3.5[deg]
 Noisy in Mach number direction
0.212
0.1090
CFD Data
CFD Data
0.1088
Linear by Adj_Grad
0.211
Quadratic by Adj_G/H
0.1086
CD
CL
Quadratic by Adj_G/H
Linear by Adj_Grad
0.210
0.1084
0.209
0.208
1.390
0.1082
1.395
1.400
Mach Number
1.405
1.410
Cl
0.1080
1.390
1.395
1.400
Mach Number
1.405
1.410
Cd
-34-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming
2D Airfoil Shape Uncertainty Analysis
Cumulative Density Function at St. Devi. of 0.01
Quadratic model only by using gradient/Hessian at optimal
Additional sampling criteria to increase total model accuracy
-35-
Yamazaki, W.,Wataru
Dept. YAMAZAKI,
of Aero. Eng.,
Tohoku
Univ.
Univ.
of Wyoming