The Importance of Adequate Verification and
Validation Strategies in Risk Management
Prof. Ch. Hirsch
Vrije Universiteit Brussel
President, NUMECA International
____________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Content


Introduction
Error identification and management
Verification requirements
Validation requirements


Beyond V&V
The new generation of simulation tools
Non-deterministic simulations
Robust design methodologies

Conclusions
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 1
Introduction

What’s new in simulation and Computer Assisted
Engineering (CAE)
CAE covers all areas of physical simulation, particularly
 Computational Structural Mechanics (CSM)
 Computational Fluid dynamics (CFD)
 Computational Electromagnetics (CEM)
Growth of computer power leads to unprecedented levels of
CAE simulations
 Million of points or degree of freedoms (DoF)
 Complex geometries
 Multiphysics (coupling fluid-thermal; fluid-structure; aeroacoustics,…)
Leading to Multidisciplinary design and optimization software
systems (MDO)
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 2
Objectives

This leads to a growing request for quality
assurance (QA) on the simulation tools

In order to respond to this request, joint efforts are
required to define:
uncertainty bounds on simulation results
limits on models
error control methodologies
appropriate test cases and QA strategies
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 3
Validation and verification



Validation: The process of determining the degree to
which a model is an accurate representation of the
real world from the perspective of the intended uses
of the model.
Verification: The process of determining that a
model implementation accurately represents the
developer's conceptual description of the model and
the solution to the model.
Verification is a prerequisite, upstream of the
validation process and requires a dedicated effort
towards adequate methodology
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 4
V&V
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 5
Identification of Error Sources

Model errors and uncertainties
Difference between exact solution of the equations and real flow
Turbulence model uncertainties predominate

Application uncertainties
Uncertain boundary conditions or geometry

Discretization or numerical errors
Difference between exact solution and solution on a finite grid
In industrial CFD, solutions are usually not grid independent

Iteration or convergence error
Difference between converged solution and result after n steps

User errors
Mistakes, blunders, carelessness, optimism, etc.,

Programming or code errors (bugs)
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 6
Verification process
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 7
Verification process



No matter how many tests and
empirical experiments we
perform, we can never prove that
a software implementation is
error free, or has zero software
defects.
Experience suggests that one can
never totally eliminate bugs from
complex CFD software, certainly
not simply by performing
algorithm testing alone.
An important approach to
algorithmic testing in verification
is the method of manufactured
solutions
From
W.L. Oberkampf, T. G. Trucano, Ch. Hirsch (2002), Verification, Validation, and Predictive Capability in Computational Engineering and Physics. Paper
presented at FOUNDATIONS 02, Foundations for Verification and Validation in the 21st Century Workshop, October 2002
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 8
Validation process
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 9
Validation hierarchy
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 10
Validation – Application
domain

In case a) we have high
confidence that the relevant
physics is understood and
modeled at a level that is
commensurate with the needs of
the application.

In case b) the boundary of the
domain indicates that outside
this region there is a degradation
in confidence in the quantitative
predictive capability of the
model.

Outside the validation domain
the model is credible, but its
quantitative capability has not be
demonstrated.
From
W.L. Oberkampf, T. G. Trucano, Ch. Hirsch (2002), Verification, Validation, and Predictive Capability in Computational Engineering and Physics. Paper presented
at FOUNDATIONS 02, Foundations for Verification and Validation in the 21st Century Workshop, October 2002
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 11
Validation – Application
domain



Figure 5a depicts the prevalent situation in engineering which shows the
complete overlap of the validation domain with the application domain. The
vast majority of modern engineering system design is represented in Fig. 5a.
Figure 5b represents the occasional engineering situation where there is
significant overlap between the validation domain and the application domain.
Some examples are: prediction of crash response of new automobile
structures, entry of spacecraft probes into the atmosphere of another planet,
and the structural response of new designs for deep-water offshore oil
platforms.
Figure 5c depicts the situation where there is no overlap between the
validation domain and the application domain.
Predictions for many high-consequence systems are in this realm because we
are not able to perform experiments for closely related conditions.
The inference from the validation domain can only be made using both
physics-based models and statistical methods. Model calibration, which
employs explicit tuning or updating of model parameters to achieve some
degree of agreement with existing validation experiments, does not fully
assess uncertainty in the predictive use of the model.

The requirement for predictive code use far from the validation database
necessitates extrapolation beyond the understanding gained strictly from
experimental validation data.
For example, the modeling of specific types of interactions of physical
processes may not have been validated together in the given validation
database. There is then uncertainty in the accuracy of model predictions
_________________________________________________________________________________________________________________________
describing such interactions.
Keynote Lecture
SIA Paris September 2003
Slide 12
Dramatic illustration

A significant example is to be found in the recent report from
NASA about the failure of the scramjet hypersonic vehicle
experiment X-43A.

The report, issued recently at http://www.space.com,
mentions:“SPACE.com has learned that the failure of the NASA X43A hypersonic aircraft in June 2001 was the result of inaccuracies
in computer and wind-tunnel tests that were based on insufficient
design information about the vehicle itself”. According to the
NASA-MIB documents, the X-43A Hyper X launch vehicle
"failed because the vehicle control system design was deficient for
the trajectory flown due to inaccurate analytical models, which
overestimated the system margins."
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 13
VERIFICATION AND VALIDATION EFFORTS
IN EUROPE
____________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
European activities

Various collective efforts in Europe for CFD
validation
ERCOFTAC: privately funded action on CFD Quality and
Trust
QNET-CFD: Thematic Network on Q&T
FLOWNET: Collection of data for validation test cases
ECORA: Q&T for CFD in the nuclear industry
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 15
PHASE 1:
ERCOFTAC BEST PRACTICE
GUIDELINES
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 16
Objectives

Guide to avoid the most common pitfalls:
Guidelines not specific to individual codes, methods or applications
Guidelines that have very wide support
Not exhaustive: 20% of rules to cover 80% of aspects

Modular form for each topic:
Discussion section
 Problem description and discussion, including references to
important books, articles and reviews with relevant examples for the
user.
 Short simple statements of advice which provide clear guidance, are
generally accepted, and are easily understandable without
elaborate mathematics.
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 17
Follow-up
Application
Data-base
Extension
Library of industrially relevant
challenging problems with quality
test data and CFD available
Extension of scope of BPG to
more difficult flow regimes
Revision
Expert
Knowledge-base
Expert knowledge on limits
of physical models in
specific flow regimes
ERCOFTAC
Best Practice
Guidelines
Updates, corrections and
improvements within
current scope
Application
Procedures
Advice on how to best do CFD and
statements on accuracy for
relevant industrial
cases
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 18
PHASE II
THE QNET-CFD NETWORK
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 19
Objectives






To assemble, structure and collate existing knowledge on the
industrial application of CFD and to make these available to
European industry
To improve the quality of the industrial application of CFD
To improve the level of trust that can be placed in industrial CFD
calculations by assembling, structuring and collating existing
know-ledge encapsulating the performance of models
underlying the current generation of CFD codes
To establish a shared database of computational and
experimental results to support industrial applications
To provide a regular state-of-the-art review on quality and trust
To promote technology transfer between industries through
workshops, regular meetings and electronic communication
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 20
Membership

44 participating organisations with representatives from :
8 European Union member states
1 EFTA member state
2 Pre-accession states




The membership is principally built from industries,
although there is a strong representation from universities
and research organisations
A diverse range of industrial sectors is represented,
including industries as diverse as civil construction and
textiles
Also included: representatives from the major European
CFD code vendors
The organisations involved range from large industrial
concerns to SME’s
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 21
Research Centres
DERA
Health and Safety Executive
NCSRD
The Meteorological Office
CEA
CIRA
CIMNE
Vattenfall Utveckling AB
Czech Academy of Science
Universities
University of Brussels (VUB)
University of Karlsruhe
University of Surrey
University of Poitiers
University of Southampton
University of Rome
Martin-Luther-Universitat
NTUA
FH Niederrhein
Cranfield University
University of Florence
University of Czestochowa
Q
N
E
T
C
F
D
Industries
WS Atkins Science & Technology
CFS Engineering SA
Renault
Sulzer Innotec
NUMECA Int.
Electricite de France
SNECMA Moteurs
MTU
AEA Technology
BMW Rolls-Royce
MAN Turbo
ABB ALSTOM Technology
Rolls-Royce Power Engineering
Fluent Europe
Magnox Electric
Mott MacDonald
SNPE Propulsion
HR Wallingford
Arup R&D
Computational Dynamics
British Nuclear Fuels
ABB Power UK
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 22
Work Procedure

Development of knowledge base
Develop Quality & Trust knowledge base that consist of a library of
application challenges (AC) within each of the industry sectors and
information on a series of well-documented flow regimes that underlie
these industrial applications








Identification and documentation of application challenges
Quality checks of application challenges
Identification of underlying flow regimes (UFR)
Documenting underlying flow regimes and development of best
practice advice
Quality checks on underlying flow regimes and review of best
practice advice
Development of application challenge best practice advice
Review of best practice advice
Exploitation of knowledge base after end of project
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 23
Quality Control





Quality Control is a major element through the
QNET-CFD operation
Managed by a “Quality Coordinator” (A. Hutton)
and a “Scientific Coordinator” (W. Rodi)
A Quality procedure document has been
generated, providing guidelines for acceptance or
rejection of submitted AC’s
All AC’s have been screened and cross-reviewed
by 2 partners
After acceptance, they are recorded in the
knowledge and data base.
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 24
Thematic Areas

The Network is organised around 6 Thematic
Areas (TA) aligned with the following industrial
sectors :
TA 1 : EXTERNAL AERODYNAMICS
TA 2 : COMBUSTION AND HEAT TRANSFER
TA 3 : CHEMICAL PROCESS, THERMAL HYDRAULICS
AND NUCLEAR SAFETY
TA 4 : CIVIL CONSTRUCTION & HVAC
TA 5 : ENVIRONMENT
TA 6 : TURBOMACHINERY INTERNAL FLOWS

Each Thematic Area has a TA coordinator that
assumes responsibility for the activities of the
group
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 25
Underlying flow regimes –
UFR 
Subdivided in 4 categories
UFR 1 -- Free flows
UFR 2 -- Flow around bodies
UFR 3 -- Semi-confined flows
UFR 4 -- Confined flow

Each UFR can be attributed to several AC’s

Contains experimental data and a variety of CFD
simulations, from which Best Practice Advice
(BPA) guidelines are derived

From the collection of BPA’s of all the UFR’s
associated to an AC, a BPA for the AC is to be
established
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 26
UFR 1 -- Free flows

1-01
Underexpanded jet
Partner-07

1-02
Blade tip and tip clearance vortex flow
Partner-09

1-03
Buoyant plumes

1-04
Annular coaxial jets, flow and mixing
Partner-12
Partner-19

1-05
Jet in a cross flow
Partner-36
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 27
UFR 2 -- Flow around bodies









2-01
Blunt base flow (streamwise flow past
cylinder with cut-off end)
2-02
Flow past cylinder
2-03
Flow around oscillating airfoil
2-04
Flow around (airfoils and) blades (subsonic)
2-05
Flow around airfoils (and blades) A-airfoil
(Ma=0.15, Re/m=2x106)
2-06
Flow around (airfoils and) blades (transonic)
2-07
3D flow around blades
2-08
Flow around grid bars
2-09
Rotor/stator interaction
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 28
UFR 3 -- Semi-confined flows

3-01


3-02
3-03
3-04
3-05
3-07

3-08

3-09
3-10
3-11
3-12
3-13
3-14
3-15
3-16










Boundary layer interacting with wakes under adverse
pressure gradient - NLR 7301 high lift configuration
Atmospheric boundary layer with rough wall
2D Boundary layers with pressure gradients
Laminar-turbulent boundary layer transition
Shock/boundary-layer interaction (on airplanes)
Natural and mixed convection boundary layers on vertical
heated walls
3D boundary layers under various pressure gradients,
including adverse pressure gradient causing separation
Impinging jet
Plane wall jet
Plane wall jet in counter current flow
Stagnation point flow
Flow over isolated hills/valleys (without dispersion)
Flow over surface-mounted cube/rectangular obstacles
2D flow over backward facing step
Wave-driven flow in a basin
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 29
UFR 4 -- Confined flow













4-01
Secondary flow in rotating and non-rotating
channels
4-02
Confined coaxial swirling jets
4-03
Pipe flow - rotating
4-04
Pipe flow - non rotating
4-05
Curved duct/pipe flow (accelerating)
4-06
Swirling diffuser flow
4-07
Developing channel flow with mass injection
through wall
4-08
Orifice/deflector flow
4-09
Confined buoyant plume
4-10
Natural convection in simple closed cavity
4-11 Simple room flow
4-12
Flows in chambers with multiple inlet/outlets
4-13
Compression of vortex in cavity
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 30
Best Practice Advices

To be derived from the BPA of individual UFR
BPA to be strongly supported by the evidence examined in the
UFR document Section on “Comparison of CFD Calculations
with Experiments”
Recommendations for Future Work
Such as new experiments to be undertaken for which the values
of key parameters are much closer to those encountered in real
application challenges
Visit http://www.qnet-cfd.net
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 31
THE FUTURE OF V&V
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 32
New approach to V&V



It is generally recognized that, although actions towards the
reduction of simulation uncertainties are still required, numerous
sources of uncertainties will always remain.
Therefore new methodologies are required in order to
incorporate the presence of uncertainties in the simulation
process and in order to introduce the existence of these
“uncertain” simulation results in the decision process related to
industrial design.
This implies responses to the following questions
How to manage uncertainties in simulations, that is
how to quantify, in a rational way, the impact of the
different sources of uncertainties on the simulation
results.
Given the existence of “uncertain” simulation results,
how are multidisciplinary design optimizations (MDO)
techniques to be developed and applied in order to
support the industrial design process?
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 33
Next steps
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 34
Required steps – 1

Quantification
uncertainties
and
management
of
simulation
Quantification and management of uncertainties is required
at the individual discipline level (CFD, CSM, CHT,…) as
well as at the integrated system level.
The current practice of CAE simulations is based on
deterministic parameters such as fixed boundary or initial
conditions, fixed geometry and physical properties, fixed
physical model parameters, etc.
However, these conditions are not generally known precisely and
are attached with unavoidable error levels, as listed above.
Therefore one should be able to evaluate simulation results by
incorporating these uncertainties in the simulation process, in
order to approach a rational quantification of uncertainties,
including the establishment of a confidence interval for the
simulation-based predictions.
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 35
Required steps – 1a


Quantification of uncertainties
Methods have to be developed to achieve the following
objectives
model and quantify the “parameter” uncertainties and
stochastic inputs (such as initial conditions, boundary
conditions, geometrical uncertainties)
model the “modeling” uncertainties associated with physical
models, transport properties, source terms
specification of the nature of the parameter and model
uncertainties
(interval
limits,
probability
density
distributions, fuzzy logic sets,…)
quantification of the parameter (and model uncertainties,
based on all possible sources of information, including
experimental data, analytical estimates, results from
computational processes, ….
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 36

Quantification of uncertain variables
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 37
Required steps – 1b

Management of uncertainties

Implement probabilistic simulations, for which innovative
mathematical and algorithmic methods have to be
developed for the treatment of differential equations
containing stochastic input parameters and model
parameters. Several methods are to be developed,
assessed and validated, such as
perturbation techniques whereby the variables of the problem are
expanded in terms of Taylor series around their mean value
(mainly for Gaussian or weakly non-Gaussian processes)
Monte Carlo methods for complete statistics, but generally
excessively expensive
Fuzzy logic methods. Fuzzy logic allows to create models based
on inexact, incomplete, or unreliable knowledge or data, and,
moreover, to infer approximate behavior of the system from such
models
Polynomial Chaos methods, whereby the randomness of the
solution uncertainties is represented and the PDF of the different
solution components is defined at every point and time.
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 38
Management of
uncertainties

Examples of
probabilistic output
for a model equation,
with a relative
viscosity variation
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 39
Probabilistic Design and Risk
management
Traditional
New non-deterministic approach
Introduction
of probabilistic
simulations
into the design
and decision
process
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 40
Probabilistic Design approach

The objectives are therefore
To develop aerodynamic and structural optimization
algorithms that provides designs, which are robust with
respect to uncertainties in geometry, operating conditions,
and code simulation uncertainties.
To control and reduce risks by providing designs with
performances insensitive to intrinsically uncertain quantities
To reduce system risks by enabling uncertainty
quantification and design strategies at the conceptual
design stage
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 41
Probabilistic Design approach

The two major classes of
uncertainty-based design
problems are robust design
problems and reliability-based
design problems.
A robust design problem seeks a
design that is relatively insensitive
to small changes in the uncertain
quantities.
A reliability-based design seeks a
design that has a probability of
failure that is less than some
acceptable value.
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 42
Conclusions

Classical verification and validation require a
large scale effort, which has to continue, although
we will never eliminate uncertainties.

Risk management needs new generation methods
to allow for
The quantitative assessment and management of
simulation uncertainties by Non-Deterministic
methodologies
Efficient probabilistic simulation outputs to quantify
reliability bounds of the predictions (mean and standard
deviations of relevant design quantities) in a rational way,
The development and application of design methodologies
incorporating probabilistic based simulations
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 43
Conclusions

The non-deterministic approach is critical in
order to
Reduce system risks by enabling uncertainty
quantification.
Increase design confidence and safety by measuring and
controlling uncertainty in performance predictions and to
enable estimates of probabilities of failure
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 44
Thank you for your attention !
_________________________________________________________________________________________________________________________
Keynote Lecture
SIA Paris September 2003
Slide 45