Keynote Tutorial: Model quality, testing and validation
Keynote Tutorial:
Methods for Testing and Validation of
Simulation Models for Engineering
Applications
David J. Murray-Smith,
School of Engineering, Rankine Building,
University of Glasgow,
Scotland, U.K.
E-mail: david.murray-smith@glasgow.ac.uk
0
Keynote Tutorial: Model quality, testing and validation 1
The presentation
1. Introduction
2. Model testing and validation issues
Uncertainties, modelling errors, testing, verification and validation.
Model quality measures and model improvements.
3. Model management
Libraries of sub-models and development of generic models.
Model version control and model documentation.
4.
Educational implications
5.
Examples
External validation and upgrading of linear and nonlinear
helicopter models for handling qualities and flight control studies.
6. Discussion and Conclusions
Keynote Tutorial: Model quality, testing and validation 2
The purpose of models in engineering
• To provide a basis for design.
• To assist in human decision making.
• To explain complex system behaviour.
• For use within fault detection systems etc..
• For simulator development (e.g. for operator training
or for engineering development applications).
Keynote Tutorial: Model quality, testing and validation 3
Design benefits with fit-for-purpose
models
1. Conceptual models allow investigation of performance
limitations at an early stage of the design process for normal
and abnormal operating conditions.
2. Fully developed and proven models can provide information
about key parameter sensitivities and inter-dependencies –
useful for design decisions and optimisation.
3. Full models allow virtual prototypes to be created before
any hardware prototype is available so identifying necessary
design alterations at an early stage, avoiding expensive
changes later on.
Keynote Tutorial: Model quality, testing and validation
4
Models in design: some pointers
 “Improved modelling of physical and manufacturing
processes will improve our ability to predict the behaviour,
costs and risks of future systems, and dramatically reduce
the development timescale”. UK Office of Science and
Technology, Technology Foresight Panel Report (1995).
 “Verification, validation and accreditation” (VV&A).
 “Smart procurement” methods.
 Concept of “the model as a specification” (as promoted by
T.S. Ericsen, US Office of Naval Research).
Keynote Tutorial: Model quality, testing and validation 5
Levels of model quality
 Level of model quality necessary is of critical importance for
any given application ........ an inappropriate model is less than
useless as it may delay the project and lead to cost escalation
 Balance needs to be found between model accuracy and the
cost of developing the model.
 Rigorous consideration of model quality is most common in
applications involving safety critical issues (e.g. aeronautical
engineering, automotive engineering etc.)
Keynote Tutorial: Model quality, testing and validation 6
Current problems with models
Models are often used with very little systematic testing.
Model documentation is often minimal and is not recognised
as a vital part of the model development process.
In many organisations models are passed from project to
project and end up being used in ways that were never
intended by the original developer of the model.
Keynote Tutorial: Model quality, testing and validation 7
Example: Hydro-turbine system
modelling
.
Figure from Bryce, G.W., Foord, T.R., Murray-Smith , D.J.
and Agnew, P., ‘Hybrid simulation of water turbine
governors’, Simulation Councils Proceedings, 6(1), 35-44,
1976
Keynote Tutorial: Model quality, testing and validation 8
Photographs: © D. J. Murray-Smith
Keynote Tutorial: Model quality, testing and validation 9
Pipeline geometry
Figure from Bryce, G.W., Foord, T.R., Murray-Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings, 6(1), 35-44, 1976
Keynote Tutorial: Model quality, testing and validation
First simplification
Figure from Bryce, G.W., Foord, T.R., Murray-Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings, 6(1), 35-44, 1976
10
Keynote Tutorial: Model quality, testing and validation
Model comparisons
(((Real-time approximation: continuous line
Non-real-time model: dashed line)
Figure from Bryce, Fo Murray-Smith and
Agnew,
Simulation Councils Proceedings, Vol.6, No. 1,
35-44.
Figure from Bryce, G.W., Foord, T.R., Murray-Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings, 6(1), 35-44, 1976
11
Keynote Tutorial: Model quality, testing and validation
12
Second approximation F
Figure from Bryce, G.W., Foord, T.R., Murray-Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings, 6(1), 35-44, 1976
Keynote Tutorial: Model quality, testing and validation
Model comparisons for second
approximation
( Real-time approximation: continuous line
Non-real-time model: dashed line)
Figure from Bryce, G.W., Foord, T.R., Murray-Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings, 6(1), 35-44, 1976
13
Keynote Tutorial: Model quality, testing and validation
Site test and simulation comparisons
Figure from Bryce, G.W., Foord, T.R., Murray-Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings, 6(1), 35-44, 1976
14
Keynote Tutorial: Model quality, testing and validation
Model predictions
15
Keynote Tutorial: Model quality, testing and validation 16
Part 2: Issues of model testing and validation
a) Model quality, uncertainties and modelling errors.
b) Testing, verification and validation of models.
c) External validation methods.
d) Model quality measures in external validation.
Keynote Tutorial: Model quality, testing and validation
17
Testing of models:
fitness-for-purpose
 Tests only deal with a small number of cases.
 General statements about validity are impossible.
 All testing must be carried out in the context of the
application and especially the precise range of operating
conditions for that application.
 Should start from a well-understood case, even if much
simplified; then move incrementally to testing for less certain
situations for that application.
 The more complex the model the harder the problem of quality
assessment becomes: measures of model performance become
harder to define and visualisation becomes more difficult.
Keynote Tutorial: Model quality, testing and validation
18
Aspects of model quality
 Establishing the useful range of the dynamic model for a
specific application.
 Estimating the limits of accuracy of the model (usually both
for steady-state and transient conditions) in terms the
magnitude of expected errors in model predictions.
Keynote Tutorial: Model quality, testing and validation 19
Sources of errors and uncertainties
• Incorrect modelling assumptions
• Errors in a priori information (e,g, model
parameter values)
• Errors in numerical solutions of model equations
• Errors in experimental procedures and in the
measurements used for model testing.
Keynote Tutorial: Model quality, testing and validation
Internal verification and
external validation
 Internal Verification – establishing that a
computer–based model is consistent with the
underlying mathematical model.
i.e. “Is the simulation model right?”
 External Validation – demonstrating that a final
(nonlinear)) model is adequate for the intended
application.
i.e. “Is it the right simulation model?”
Note that checks of identified linear models are
sometimes referred to as “external verification”.
20
Keynote Tutorial: Model quality, testing and validation
External validation
Need to distinguish between:
 Functional validation: where model is assessed in
terms of how well it mimics input-output behaviour
of the real system.
 Physical/Theoretical validation where the model
is based on theory and intermediate variables in
model and system are compared. Approximations
and assumptions are investigated within this
process.
21
Keynote Tutorial: Model quality, testing and validation
Approaches to external validation
 Holistic approaches (e.g. subjective opinion of an
expert on the real system such as an operator).
 Model component approaches (e.g. each subsystem tested independently and compared with
corresponding components of the real system).
22
Keynote Tutorial: Model quality, testing and validation
23
Methods for external validation
 Methods involving direct comparisons of response
data from model and real system.
 Methods based on system
parameter estimation.
identification
and
 Methods involving parameter sensitivity analysis.
 Methods based on inverse models and inverse
simulations.
Whatever method is used, data for external validation
must be appropriate for the intended application.
Careful experimental design is essential.
Keynote Tutorial: Model quality, testing and validation 24
Methods for system and model
comparisons
 Graphical comparisons
 Quantitative measures
 System identification methods
 Expert opinion
Different approaches can provide different kinds of
insight.
Keynote Tutorial: Model quality, testing and validation
25
Methods of system/model
comparison: some examples
 Plots of simulated and measured responses against an independent
variable (often time).
 Plots of simulated values against the corresponding measured values
(should be 45 degree line).
 Different graphical methods may emphasise different aspects of the
simulation model performance so there are possible benefits from
combining different approaches.
Keynote Tutorial: Model quality, testing and validation
Typical time history comparisons
26
Keynote Tutorial: Model quality, testing and validation
27
Another time history comparison: a
multi-input multi-output case
BO-105 helicopter flight test data,
DLR SIMH simulation model
The original version of this
figure was published by the
Advisory Group for Aerospace
Research and Development,
North Atlantic Treaty
Organisation (AGARD/NATO)
in AGARD Advisory Report
280 ‘Rotorcraft System
Identification’, September
1991
Keynote Tutorial: Model quality, testing and validation
Examples of simple cost functions
Mean absolute error
Mean absolute percent error
Weighted error
28
Keynote Tutorial: Model quality, testing and validation 29
Theil’s Inequality Coefficient (TIC)
Keynote Tutorial: Model quality, testing and validation
Use of quantitative measures:
an example
30
Keynote Tutorial: Model quality, testing and validation
31
Polar diagram form of display
From Smith, M.I., Murray-Smith, D.J. and Hickman,
D., ‘Verification and validation issues in a generic
model of electro-optic sensor systems ‘ J. Defense
Modeling and Simulation, 4(1), 17-27, 2007
Keynote Tutorial: Model quality, testing and validation 32
Issues of identifiability in external
validation
Test input design is important since inputs must excite the
system and model over an appropriate range of frequencies and
amplitudes.
The concept of identifiability is central to issues of test input
design external validation and is thus very important for external
validation.
Structural identifiability relates to situations where a model
may have an excess of parameters so that some specific
parameters cannot be estimated uniquely for any possible
experimental design (e.g. Bellman, R. and Åström, K.J.,
Mathematical Biosciences, 7, 329-339, 1970). Structural
identifiability is also important for external validation.
Keynote Tutorial: Model quality, testing and validation
33
Issues of numerical identifiability and
test input design in model validation
Numerical unidentifiability arises when a structurally identifiable
model is being used with data that is inappropriate for the application.
Numerical identifiability investigated from parameter information
matrix M, the related dispersion matrix D and the parameter correlation
matrix P. All depend on the sensitivity matrix X where:
pij 
mij 1
mii 1m jj 1
Inputs may maximise the overall accuracy of all parameter estimates or
may be chosen to maximise accuracy of specific parameter estimates.
Keynote Tutorial: Model quality, testing and validation
Upgrading of simulation models
Following comparison of model and system behaviour usually
need to analyse discrepancies and propose upgrades for model.
Changes must be evaluated systematically on a physical basis –
with further iterations in the development cycle.
Parametric changes usually considered first, before structure .
Sometimes possible to associate model deficiencies with specific
state variables model (e.g. correlation of output error with a state
variable may help identify problem source).
Correlation of model errors with derivatives of state variables
may suggest that a higher-order description would be more useful.
Optimisation tools (including evolutionary computing methods
such as GA and GP) may be useful but should be used along with
physical knowledge and understanding of the model.
34
Keynote Tutorial: Model quality, testing and validation
35
Part 3: Model management
a) Libraries of sub-models and generic models.
b) Model version control and model documentation.
Keynote Tutorial: Model quality, testing and validation 36
Generic and re-usable sub-models
 Generally accepted that system design should be based on use of
generic descriptions and re-usable sub-models.
 Examples of the generic approach may be found in automotive
engineering, gas turbines etc. Issues inevitably arise in the external
validation of generic models – one approach is discussed in Smith,
M.I., Murray-Smith, D.J. and Hickman, D., ‘Verification and
validation issues in a generic model of electro-optic sensor systems’
J. Defense Modeling and Simulation, 4(1), 17-27, 2007
Keynote Tutorial: Model quality, testing and validation
37
Model documentation and version control
 Extra costs of creating good documentation should be
more than balanced by the resulting re-usability of
models. Version control processes should ensure that
changes are fully documented.
 Documentation and version control well developed in
software engineering field. Same principles should be
applied to the model development process.
Keynote Tutorial: Model quality, testing and validation
Items for documentation
 Purpose of model and the intended application.
 Full description of model and corresponding computer
code.
 List of all assumptions and approximations used.
 Details of all tests carried out on the real system to provide
information for model development.
 Details of the internal verification process.
 Details of the external validation process, with statements
about why model was accepted or rejected and information
about usable range for model.
38
Keynote Tutorial: Model quality, testing and validation
Part 4: Educational implications
39
Keynote Tutorial: Model quality, testing and validation
40
Modelling and simulation in
engineering education
 Engineering students encounter mathematical and computerbased modelling repeatedly in their university education.
 Emphasis is most often on development of models from
physical principles and on using models/simulations in place
of experiments on real systems.
Keynote Tutorial: Model quality, testing and validation 41
Model quality and testing issues in
engineering education
 Issues of model quality and fitness-for-purpose are seldom emphasised
in the teaching of modelling and simulation.
 Model validation is neglected in education. The teaching of
system modelling and simulation should include much more on
model validation methods.
 Model testing should become second nature for students.
 Documentation, model re-use and libraries of models must be
given much more emphasis (especially in more advanced teaching).
Keynote Tutorial: Model quality, testing and validation 42
Part 5: Examples
Drawn from external validation and upgrading of
helicopter models for flight control system
design.
Keynote Tutorial: Model quality, testing and validation
Model limitations in control
 Plant model limitations often impose serious
performance limitations within control system,
especially in systems with high-performance
requirements.
 Particularly important to have highly accurate plant
models for the part of the frequency range close to
the “cross-over” region in the frequency domain.
43
Keynote Tutorial: Model quality, testing and validation 44
Examples from aircraft and helicopter
flight control system design
• There are many well documented aircraft flight
control examples illustrating problems of model
quality and model limitations in integrated system
design.
• Problems are often identified during initial testing
of hardware. These lead to development of
improved models and corresponding control design
changes.
• The later in the design cycle these changes have to
be made the more costly they are and the greater
the delays to the project.
Keynote Tutorial: Model quality, testing and validation 45
Helicopter model requirements
 Must perform well over a defined
range of frequencies.
 Must perform well over a defined
range of manoeuvre amplitudes.
Photograph: © D. Murray-Smith
For flight control design model must perform especially
well close to open-loop gain and phase cross-over
frequencies (as with control applications involving other
types of system).
Nominal model: simulation and flight
data compared for same test inputs.
(Westland Lynx helicopter in 300 ft. quick-hop manoeuvre)
From Bradley, R., Padfield, G.D., Murray-Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical
models’, Transactions of the Institute of Measurement and Control’, 12(4), 186-196, 1990.
Common test inputs used for system
identification and “external verification”
of the identified model (BO-105 flight data)
SS
The original versions of these figures were published by
the Advisory Group for Aerospace Research and Development,
North Atlantic Treaty Organisation (AGARD/NATO) in AGARD
Advisory Report 280 ‘Rotorcraft System Identification’, September 1991
Keynote Tutorial: Model quality, testing and validation 49
...........and flight testing
Simulation, identification and
“external verification” results
(BO-105 flight test data, DLR SIMH model)
The original version of these figures
were published by the Advisory Group
for Aerospace Research and
Development, North Atlantic Treaty
Organisation (AGARD/NATO) in
AGARD Advisory Report 280
‘Rotorcraft System Identification’,
September 1991
Estimated and theoretical parameter values of the
identified model for lateral/directional
characteristics
(SA-330 Puma helicopter flight test data : 80 knots straight and level flight)
L
p
N
p
L
r
N
v
L
v
N
r
Y
v
L
δ lat
Normalised HELISTAB
values
2ζω0
N
N
δ lat
L
Values estimated from
flight data using system
identification methods.
δ ped
δ ped
Keynote Tutorial: Model quality, testing and validation 53
Assessment of a theoretical
nonlinear model for a Puma helicopter
Parameter values for two
different flight conditions,
showing parametric trends from
a physically-based nonlinear
simulation model (HELISTAB)
and the trends in estimates from
flight tests involving system
identification of separate
linearised models for each
flight condition.
From Bradley, R., Padfield, G.D., Murray-Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical models’, Transactions of
the Institute of Measurement and Control’, 12(4), 186-196, 1990.
Keynote Tutorial: Model quality, testing and validation 55
Summary of model quality issues for helicopter
flight control system design
Good vehicle models are essential for design of high-bandwidth
full-authority active flight control systems.
Published examples show that the achievable performance of flight
control systems have, in some cases, been overestimated in initial
design studies, usually because of limitations of the flight mechanics
model of the vehicle (see e.g. Tischler, M. B. Advances in Aircraft Flight
Control Systems, Taylor & Francis, London 1996).
Although control systems can be made robust to compensate for poor
accuracy this is usually at the expense of performance. Improved modelling
procedures and improved models can offer significant benefits. Otherwise,
problems may not be apparent until the flight testing stage, leading to
costly redesign, extended flight test programmes and delays in certification.
Keynote Tutorial: Model quality, testing and validation 56
Part 5: Discussion and conclusions
Keynote Tutorial: Model quality, testing and validation
57
Discussion: Model quality in design
 The more demanding the design specification the more important is the
fitness-for-purpose of models used in design.
 More attention needs to be given to the external validation of models
for the specific application in question.
a) Establishing the useful range of the model.
b) Estimating the accuracy of the model within that range.
 Model validation is part of the model building process and external
validation techniques need to be applied repeatedly.
 Models should be retained, maintained and updated throughout the
whole life-cycle of the system that they represent.
Keynote Tutorial: Model quality, testing and validation 58
Recommendations
• More attention should be given to the fitness-for-purpose of models
used in design, especially for demanding applications.
• Current methods for external validation are time-consuming and
difficult to apply in many situations. More effort should be devoted to
improving validation methods.
• Techniques of version control and rigorous documentation should
borrowed from software engineering and applied to the model
development process. Re-use of proven models should be made easier
and more comprehensive model documentation should be available
within model libraries.
• Issues of model quality should be given far more attention within
engineering education.
Keynote Tutorial: Model quality, testing and validation 59
Conclusions
With suitable structure and parameter values and rigorous
external validation models that are fit-for-the-purpose of
a given application can be developed (iteratively of course).
Good model management can reduce the cost of design
and development.
There are no quick answers: a systematic approach is
essential, moving incrementally from well-understood
cases to less well known situations.
Educational and cultural changes are needed as well as
improved management of the modelling, simulation and
design processes within most organisations.