Characterization and Prediction of CFD Simulation Uncertainties
PhD Preliminary Oral Exam
CHARACTERIZATION AND PREDICTION OF CFD SIMULATION
UNCERTAINITIES
by
Serhat Hosder
Chair: Dr. Bernard Grossman
Committee Members:
Dr. Raphael T. Haftka
Dr. William H. Mason
Dr. Reece Neel
Dr. Rimon Arieli
Department of Aerospace and Ocean Engineering
Virginia Tech.
Blacksburg, VA
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
0
Characterization and Prediction of CFD Simulation Uncertainties
Outline of the Presentation
•
•
•
•
•
•
Introduction
Classification of CFD Simulation Uncertainties
Objective of the Present Work
Previous Studies
Transonic Diffuser Case
Results, findings and discussion about different sources
of uncertainty
• Conclusions
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
1
Characterization and Prediction of CFD Simulation Uncertainties
Introduction (1)
• The Computational Fluid Dynamics (CFD) as an
aero/hydrodynamic analysis and design tool
• Increasingly being used in multidisciplinary design and
optimization (MDO) problems
• Different levels of fidelity (from linear potential
solvers to RANS codes)
• CFD results have a certain level of uncertainty
originating from different sources
• Sources and magnitudes of the uncertainty
important to assess the accuracy of the results
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
2
Characterization and Prediction of CFD Simulation Uncertainties
Introduction (2)
Drag Polar Results for DLR F-4 Wing at M=0.75, Rec=3x106 (taken from 1st AIAA Drag
Prediction Workshop (DPW), Ref. 1)
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
3
Characterization and Prediction of CFD Simulation Uncertainties
Classification of CFD Simulation Uncertainties
• Physical Modeling Uncertainty
– PDEs describing the flow (Euler, Thin-Layer N-S, Full N-S, etc.)
– Boundary and initial conditions (B.C and I.C)
– Auxiliary physical models (turbulence models, thermodynamic
models, etc.)
• Uncertainty due to Discretization Error
– Numerical replacement of PDEs and continuum B.C with
algebraic equations
– Consistency and Stability of PDEs
– Spatial (grid) and temporal resolution
• Uncertainty due to Iterative Convergence Error
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
4
Characterization and Prediction of CFD Simulation Uncertainties
Definition of “Uncertainty” and “Error”
• Oberkampf and Trucano (Ref. 2) defined
Uncertainty as a potential deficiency in any phase or activity of
modeling process that is due to the lack of knowledge (uncertainty of
turbulence models, geometric dimensions, thermo-physical
parameters, etc.)
Error as a recognizable deficiency in any phase or activity of
modeling and simulation
• Discretization errors can be estimated with certain methods by
providing certain conditions
• In this work, we’ll refer the inaccuracy in the CFD simulations due
different sources as “uncertainty”
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
5
Characterization and Prediction of CFD Simulation Uncertainties
Objective of the Present Work
• Characterize different sources of CFD simulation
uncertainties
– Consider different test cases
– Apply different grids, solution schemes/parameters, and physical
models
• Try to quantify/predict the magnitude and the relative
importance of each uncertainty
• Compare the magnitudes of CFD simulation uncertainties
with other sources of uncertainty (geometric uncertainty,
uncertainty in flow parameters, etc.)
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
6
Characterization and Prediction of CFD Simulation Uncertainties
Previous Studies
• Previous CFD related studies mainly focused on
discretization and iterative convergence error estimations
– Grid Convergence Index (GCI) by Roache (Ref. 3)
– Discretization Error of Mixed-Order Schemes by C. D. Roy (Ref. 4)
• Trucano and Hill (Ref. 5) proposed statistical based
validation metrics for Engineering and Scientific Models
• Hemsch (Ref. 6) performed statistical analysis of CFD
solutions from 1st AIAA DPW.
• Kim (Ref. 7) made statistical modeling of simulation errors
(from poorly converged optimization runs) and their
reduction via response surface techniques
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
7
Characterization and Prediction of CFD Simulation Uncertainties
Description of Transonic Diffuser Test Case (1)
• Known as “Sajben Transonic Diffuser” case in CFD Validation studies
• Top wall described by an analytical equation
• Although geometry is simple, the flow-field is complex.
• The Shock strength and the location determined by exit-pressure-to-inlettotal pressure ratio Pe/P0i
• Pe/P0i=0.72 (Strong shock case), Pe/P0i=0.82 (Weak shock case),
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
8
Characterization and Prediction of CFD Simulation Uncertainties
Description of Transonic Diffuser Test Case (2)
Mach contours for the weak shock case
Mach contours for the strong shock case
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
9
Characterization and Prediction of CFD Simulation Uncertainties
Simulation Code, Solution Parameters, and Grids (1)
• General Aerodynamic Simulation Program (GASP)
– 3-D, structured, multi-block, finite-volume, RANS code
• Inviscid fluxes calculated by upwind-biased 3rd (nominal)
order spatially accurate Roe-flux scheme
• All viscous terms were modeled (full N-S)
• Implicit time integration to reach steady-state solution
with Gauss-Seidel algorithm
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
10
Characterization and Prediction of CFD Simulation Uncertainties
Simulation Code, Solution Parameters, and Grids (2)
• Flux-Limiters
– Van Albada’s limiter
– Min-Mod limiter
• Turbulence Models
– Spalart-Allmaras (Sp-Al)
– k-w (Wilcox, 1998 version)
• Grids Generated by an algebraic mesh generator
–
–
–
–
–
Grid 1 (g1): 41x26x2
Grid 2 (g2): 81x51x2
Grid 3 (g3): 161x101x2
Grid 4 (g4): 321x201x2
Grid 5 (g5): 641x401x2 (Used only for Sp-Al, Min-Mod,
strong shock case)
• y+= 0.53 (for g2) and y+= 0.26 (for g3) at the bottom wall
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
11
Characterization and Prediction of CFD Simulation Uncertainties
Output Variables (1)
Nozzle efficiency, neff
neff
H 0i  H e

H 0 i  H es
H0i : Total enthalpy at the inlet
He : Enthalpy at the exit
Hes : Exit enthalpy at the state that would be reached by isentropic
expansion to the actual pressure at the exit
yi
H0i   (y) u(y) hoi(y) d y
0
He 
ye
  ( y) u( y) h ( y) d y
 Pe ( y ) 

hes ( y )  c pToi 
 P0 i 
 1

e
0
ye
Hes   ( y) u( y) hes ( y) d y
y
y
ht
Throat height
0
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
12
Characterization and Prediction of CFD Simulation Uncertainties
Output Variables (2)
Orthogonal Distance Error, En
A measure of error in wall pressures between the experiment and
the curve representing the CFD results
N ex p
En 
di  min xinlet  x  xexit
d
i
i 1
N exp
 ( x  x )  ( P ( x)  P
2
i
c
2
(
x
))
exp
i

Pc : Wall pressure obtained from CFD calculations
Pexp: Experimental Wall Pressure Value
Nexp: Total number of experimental points (Nexp=36)
di: Orthogonal distance from the ith experimental data point to Pc(x) curve
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
13
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to iterative convergence error (1)
• Normalized L2 Norm
Residual of the energy
equation for the case with
Sp-Al turbulence model,
Van-Albada and Min-Mod
limiters at the strong shock
case.
• Same convergence behavior with respect to the limiters observed
for the k-w case.
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
14
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to iterative convergence error (2)
Poor L2 norm convergence does not seem to effect the convergence
of the neff results at different grid levels
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
15
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to iterative convergence error (3)
Roy and Blottner (Ref. 8) proposed a method to estimate,  the
 e
iterative convergence error at time level (cycle) n
n
 tn
n
neff
 neff (t n )  (neff ) exact   n
Assuming exponential decrease for   e
n
  e
n
 tn
 tn
Need three time levels in the exponential region
(neff ) exact 
n
n 1
neff
 n neff
1 
n
n
% error of neff
where
n 
n 
n
n 1
neff
 neff
1  n
n
n 1
 neff

 neff
 100 n
n n 1 
 neff   neff 
n 1
n
neff
 neff
n
n 1
neff
 neff
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
16
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to discretization error (1)
For each case with a different turbulence model, grid level (resolution)
and the flux-limiter affect the magnitude of the discretization error
• The effect of the limiter observed at grid levels g1 and g2
• At grid levels g3 and g4, the effect is much smaller
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
17
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to discretization error (2)
• Richardson’s extrapolation method:
f k  f exact   h p  O(h p 1 )
h: a measure of grid spacing
p: The order of the method.
• Assumptions needed to use Richardson’s method:
– Grid resolution is in the asymptotic region
– The order of the spatial accuracy, p should be known. Usually
observed order of spatial accuracy is different than the nominal
value. The observed order should be determined.
– Monotonic grid convergence. Mixed-order schemes can cause
non-monotonic convergence. Roy (Ref. 4) proposed a method for
for the discretization error estimate of mixed-order schemes.
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
18
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to discretization error (3)
turb.
model
limiter
Pe/P0i
p
(neff)exact
Sp-Al
Van Albada
0.72
1.528
0.71830
Sp-Al
Sp-Al
Sp-Al
k-w
k-w
Min-Mod
Van Albada
Min-Mod
Van Albada
Min-Mod
0.72
0.82
0.82
0.82
0.82
1.322
1.198
1.578
1.980
1.656
0.71590
0.80958
0.81086
0.82962
0.82889
grid
level
g1
discretization
error (%)
9.820
g2
4.505
g3
1.562
g4
0.542
g1
14.298
g2
6.790
g3
2.716
g4
1.086
g1
6.761
g2
3.507
g3
1.528
g4
0.666
g1
8.005
g2
3.539
g3
1.185
g4
0.397
g1
3.514
g2
1.459
g3
0.370
g4
g1
0.094
4.432
g2
1.452
g3
0.461
g4
0.146
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
19
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to discretization error (4)
• “p” values are dependent on the
grid levels used
• However the difference between
the (neff)exact values are small
compared to overall uncertainty
turb.
model
Sp-Al
Sp-Al
limiter
Min-Mod
Min-Mod
Pe/P0i
0.72
0.72
grid
levels
used
g2, g3,
and g4
g3, g4,
and g5
p
1.303
2.026
neff_exact
0.715235
0.719492
grid
level
discretization
error (%)
g1
14.405734
g2
6.940532
g3
2.812698
g4
1.139865
g5
0.729023
g1
13.728739
g2
6.307712
g3
2.204303
g4
0.5413695
g5
0.1329585
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
20
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to discretization error (5)
• The uncertainty due to discretization error is bigger for
the cases with strong shock compared to the weak shock
results at each grid level. The flow structure has
significant effect on the discretization error.
• For the monotonic cases, largest errors occur for the SpAl, Min-Mod, strong shock case and the smallest errors
are obtained for the k-w , Van-Albada, weak shock case
• Non-monotonic convergence behavior for the cases with
k-w and the strong shock as the mesh is refined
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
21
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to discretization error (6)
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
22
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to discretization error (7)
• Noise due to discretization error observed at grid levels
1 and 2.
• Noise error small compared to the systematic
discretization error between each grid level. However,
this can be important for gradient-based optimization.
• Kim (Ref. 7) successfully modeled the the noise error
due to poor convergence of the optimization runs by
fitting a probability distribution (Weibull) to the error.
• The noise error can be reduced via response surface
modeling.
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
23
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to turbulence models (1)
• Uncertainty due to turbulence modeling (in general
physical modeling) should be investigated after
estimation of the discretization and iterative
convergence error.
• Difficult to totally separate physical modeling errors
from discretization errors
• “Validation” of the Engineering and Scientific Models
deals with accuracy of the physical model
• Need high-quality experimental data
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
24
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to turbulence models (2)
• Orthogonal distance error, En is used for comparison
of CFD results with the experiment
En for each case is scaled with the maximum value
obtained for k-w , Min-Mod, strong shock case
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
25
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to turbulence models (3)
For each case (strong shock or weak shock), best match with the experiment
is obtained with different turbulence models at different grid levels
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
26
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to turbulence models (4)
• Experimental uncertainty should be considered
• With the experimental geometry, a perfect match with CFD and
experiment can be observed upstream of the shock
• Upstream of the shock, discrepancy between CFD simulations and
experiment is most likely due to the experimental uncertainty
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
27
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to turbulence models (5)
• A better way of using En for this example would be to
evaluate it only downstream of the shock
• The discretization and iterative convergence error should
be estimated for En in a similar way used for the nozzle
efficiency
• An estimate of exact value of (En ) can be used for
approximating the uncertainty due to turbulence models
• The relative uncertainty due to the selection of
turbulence models can also be investigated by using
(neff)exact values obtained by Richardson’s extrapolation
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
28
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to turbulence models (6)
• Hills and Trucano (Ref. 5) proposed a Maximum Likelihood based
model validation metric to test the accuracy of the model predictions
• Uncertainty in the experimental measurements and the model
parameters are considered
– Model parameters:
• Material properties
• Geometry
• Boundary or Initial Conditions
• This method requires prior knowledge about the measurement and
the model parameter uncertainty (modeling with probabilistic
distributions)
• Looks for statistically significant evidence that the model validations
are not consistent with the experimental measurements
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
29
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to turbulence models (7)
• PDF(d) : PDF of measurement vector occurrence
PDF(p) : PDF of model parameter vector occurrence
• PDF(d, p) = PDF(d) x PDF(p)
• Find the maximum likely values for the mode of the measurements
d and the model parameters p
– Find the maximum value of Joint PDF via optimization
• Evaluate the probability of obtaining a smaller PDF assuming that
the model is correct
• If this value is bigger than the level of significance that we assumed
for rejecting a good model, than the model predictions are
consistent with the experiment
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
30
Characterization and Prediction of CFD Simulation Uncertainties
Uncertainty due to turbulence models (8)
•
Possible application to test the accuracy of the turbulence
models
– Takes into account the experimental uncertainty
– Requires prior knowledge of uncertainty in the measurements and
the model parameters
– Selection of model parameters
– No simple relationship with the model parameters and the output
quantities. Using response surface techniques may be needed to
find a functional form.
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
31
Characterization and Prediction of CFD Simulation Uncertainties
Additional Test Cases
• Need more cases to generalize the results obtained in Transonic
Diffuser Case
• Next possible case : Steady, turbulent, flow around an airfoil
(RAE2822 or NACA0012)
–
–
–
–
Consider transonic and subsonic cases
Consider a range of AOA
Output quantities to monitor: Cl, Cd, Cp distributions
Orthogonal distance error may be used for characterizing Cp
distributions
• Consider a case with a more complex geometry
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
32
Characterization and Prediction of CFD Simulation Uncertainties
Conclusions (1)
• Different sources of uncertainty in CFD simulations should be
investigated separately.
• Discretization and iterative convergence errors can be estimated by
certain methods in certain conditions
• Limiters affect the iterative convergence and the discretization error.
– L2 norm convergence affected by the use of different limiters
– Poor L2 norm convergence do not seem to affect the neff results
• Asymptotic Grid convergence hard to obtain
• Flow structure has a strong effect on the magnitude of the
discretization error.
• Iterative convergence error small compared to the discretization error
• Uncertainty due to turbulence model should be investigated after the
estimation of discretization and iterative convergence error.
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
33
Characterization and Prediction of CFD Simulation Uncertainties
Conclusions (2)
• Comparison with the experiment is needed to determine the
accuracy of the turbulence models
• Experimental uncertainty should be considered possibly by using a
statistical method
• More cases need to be analyzed to generalize the results
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
34
Characterization and Prediction of CFD Simulation Uncertainties
References
1.
2.
3.
4.
5.
6.
7.
8.
Levy, D. W., Zickuhr, T., Vassberg, J., Agrawal S., Wahls. R. A., Pirzadeh, S., Hemsch, M. J.,
Summary of Data from the First AIAA CFD Drag Prediction Workshop, AIAA Paper 20020841, January 2002
Oberkampf, W. L. and Trucano, T. G., Validation Methodology in Computational Fluid
Dynamics. AIAA Paper 2000-2549, June 2000
Roache, P. J. Verication and Validation in Computational Science and
Engineering.Hermosa Publishers, Albuquerque, New Mexico, 1998.
Roy, C. J., Grid Convergence Error Analysis for Mixed-Order Numerical Schemes,
AIAA Paper 2001-2606, June 2001
Hills, R. G. and Trucano, T. G., Statistical Validation of Engineering and Scientific
Models: A Maximum Likelihood Based Metric, Sandia National Loboratories,
SAND2001-1783
Hemsch, M. J., Statistical Analysis of CFD Solutions from the Drag Prediction
Workshop, AIAA Paper 2002-0842, January 2002
Kim, H., Statistical Modeling of Simulation Errors and Their Reduction Via Response
Surface Techniques, PhD dissertation, VPI&SU, June 2001
Roy, C. J. and Blottner F. G., Assesment of One-and Two-Equation Turbulence
Models for Hypersonic Transitional Flows, Journal of Spacecraft and Rockets,
Vol.38, No. 5, September-October 2001
Ph.D Preliminary Oral Exam, Aerospace and Ocean Engineering Department, February 27th 2002
35