ABSTRACTS
FOR CONTRIBUTED TALKS
(In alphabetical order)
UCM 2012 CONFERENCE
2-4 JULY 2012
KENWOOD HALL, SHEFFIELD, UK
UCM 2012 Conference : Contributed Talk Abstracts
Page1 of 9
National Oceanography Centre, Southampton
Ioannis Andrianakis
Contributed Talk
Super parameterisation of ocean convection using emulators
Sub-grid scale processes play an important role in ocean and climate modelling. Typical examples include clouds in
atmospheric models, flows over restricted topographies or the resolution of convective plumes in ocean models.
Detailed numerical models of these sub-grid scale processes exist, but embedding them in a Global Circulation Model
(GCM) for example, would be computationally prohibitive. In the present work we investigate the applicability of
emulators for representing the sub-grid scale processes within a GCM simulation. Emulators can be thought as
encapsulating our beliefs about the sub-grid dynamical model, derived from a designed computer experiment using a
Bayesian framework. In particular, we propose to employ an emulator for parameterising the sub-grid scale process, and
embed this within a GCM as a surrogate for the actual sub-grid scale model. The result of combining the GCM with the
emulator will be a ‘super-parameterised’ model, which will also be computationally efficient, since the emulator incurs a
very small computational overhead. The example we chose to illustrate the proposed methodology is deep ocean
convection. The sub-grid scale dynamical model simulates deep convective plumes, while the large scale dynamics
simulate the geostrophic eddy scale. We present details on building the emulator of the convective plumes and its
coupling with the large scale process model.
Peter Challenor, NOCS
JOINT AUTHORS:
University of Oxford
Hannah Arnold
Contributed Talk
Reliably predicting uncertainty in weather and climate forecasts
The climate system is highly complex, and weather and climate prediction are mathematically challenging. Many
uncertainties are involved which are not just due to our lack of knowledge about future greenhouse gas emissions. The
process of representing the atmosphere, oceans and land-surface, and their many interactions, in a piece of computer
code necessarily introduces other errors. Limited computer resources dictate that code must run efficiently, and results
in further approximations and simplifications, whereby unresolved small scale processes are represented in
parametrisation schemes. A single prediction is of limited use as it gives no indication of the possible error in the
prediction. Instead, an ensemble of predictions should be generated, and the spread of the ensemble used to estimate
the uncertainty in the projection due to limitations in the forecast model.
Several methods of generating this ensemble have been proposed. The Intergovernmental Panel on Climate Change use
a multi-model ensemble: different modelling centres submit their models’ projections and the range of these is used as a
pragmatic estimate of uncertainty. However, these “ensembles of opportunity” are small in size and are not designed to
explore the range of climate projections consistent with the model error.
In our experiment, we develop a new technique involving stochastic mathematics for representing the small scales in an
atmospheric model, providing a physically motivated way of representing model uncertainty. These stochastic
parametrisations include random numbers in the representations of unresolved processes in the model. The unresolved
variables are not fully constrained by the grid-scale variables; stochastic schemes can represent this error between the
deterministic scheme and the true system and can explore the uncertainty inherent in the parametrisation process.
Several different parametrisations are tested, including the use of additive and multiplicative noise. This new technique
was compared to the more traditional approach of using perturbed parameter ensembles. These are generated by
changing the value of parameters in the deterministic parametrisation schemes. Such parameters are poorly defined due
to lack of observational data, so varying them can sample uncertainties in the deterministic parametrisation. These two
schemes were tested using an idealised mathematical model of the atmosphere where the true solution is known, which
also provides a framework for analysing their ability to represent model uncertainty reliably.
Stochastic parametrisation ensembles were found to give significantly better representation of model uncertainty than
perturbed parameter ensembles. The reliability (statistical consistency) of the short range ensemble forecasts was better
for the stochastic parametrisation schemes, indicating these ensembles give good estimates of the probability that a
particular event will occur. This characteristic of an ensemble is very important in weather forecasting, especially when
forecasting extreme weather events. An improved reliability of weather forecast was also found to result in a better
forecast climatology. This link between short-term reliability and climatological skill could be used to test climate models
and verify their projections. The best stochastic parametrisations included a temporal autocorrelation in the noise term,
indicating the importance of physically motivated stochastic schemes.
We have demonstrated the power of stochastic schemes over more traditional perturbed parameter schemes for
accurately representing model uncertainty in weather and climate simulations. We hope our results motivate the
development of stochastic climate simulators alongside the stochastic numerical weather prediction models currently
operationally in use.
JOINT AUTHORS:
UCM 2012 Conference : Contributed Talk Abstracts
Dr I. M. Moroz & Prof. T. N. Palmer
Page2 of 9
Université Paris-Sud / AgroParisTech
Pierre Barbillon
Contributed Talk
Bounding rare event probabilities in computer experiments
We are interested in bounding probabilities of rare events in the context of computer experiments. These rare events
depend on the output of a physical model with random input variables. Since the model is only known through an
expensive black box function, standard efficient Monte Carlo methods designed for rare events cannot be used. We then
propose a strategy to deal with this difficulty based on importance sampling methods. This proposal relies on Kriging
metamodeling and is able to achieve sharp upper confidence bounds on the rare event probabilities. The variability due
to the Kriging metamodeling step is properly taken into account. The proposed methodology is applied to a toy example
and compared to more standard Bayesian bounds. Finally, a challenging real case study is analyzed. It consists of finding
an upper bound of the probability that the trajectory of an airborne load will collide with the aircraft that has released it.
Keywords: computer experiments, rare events, Kriging, importance sampling, Bayesian estimates, risk assessment with
fighter aircraft.
Yves Auray & Jean-Michel Marin
JOINT AUTHORS:
Aston Unviersity, Birmingham
Alexis Boukouvalas
Contributed Talk
Optimal Design for Stochastic Emulation with Heteroscedastic Gaussian Process Models
We examine optimal design for parameter estimation of Gaussian process regression models under input-dependent
noise on the model outputs. Our motivation stems from the area of computer experiments, where computationally
demanding simulators are approximated using Gaussian process emulators as statistical surrogates. In the case of
stochastic simulators, also known as random output simulators, the simulator may be evaluated repeatedly for a given
parameter setting allowing for replicate observations in the experimental design. Our findings are applicable, however,
in the wider context of experimental design for Gaussian process regression and kriging.
Designs are proposed with the aim of minimising the variance of the Gaussian process parameter estimates, that is we
seek designs that enable us to best learn about the Gaussian process model. We construct heteroscedastic Gaussian
process models and propose an experimental design technique based on an extension of the use of Fisher information to
heteroscedastic models.
We empirically show that although a strict ordering of the Fisher information to the variance of the maximum likelihood
parameter estimates is not exact, the approximation error is reduced as the number of replicated points is increased. We
consider both local and Bayesian D-optimal designs in our experiments. Through a series of simulation experiments on
both synthetic data and a systems biology model, the replicate-only optimal designs are shown to outperform both
replicate-only and non-replicate space-filling designs as well as non-replicate optimal designs. The results enable us to
provide guidance on best practice for optimal experimental design for stochastic simulators. This work builds on our
existing framework presented in UCM 2010 by providing further insights into the validity of the Fisher information as a
design criterion for correlated processes as well an extensive simulation study on systems biology models.
Dan Cornford, Milan Stehlik
JOINT AUTHORS:
National Oceanography Centre
Peter Challenor
Contributed Talk
Emulators for Extremes Estimated From Numerical Simulators
Extreme values (floods, heat waves, giant waves, …) are of great practical importance. Often these are estimated from
simulator output; either from a hindcast (running the simulator over the past) or a forecast. At present the uncertainty
arising from the simulator is not included when estimating extremes. In this paper we examine two methods of
approaching this problem. Because of the Fisher-Tippett limits we know the form of the distribution of extremes.
Because extreme value distributions are very non-Normal the use of a Gaussian process is not appropriate. We
investigate two ways of producing emulators for extremes. The first is to use a multivariate Gaussian process from the
parameters of the extreme value distribution; the second is to use a max-stable process. We will discuss the differences
between these methods and how we should deal with the inherent stochastic nature of extremes and the sampling
variability from the simulator.
JOINT AUTHORS:
UCM 2012 Conference : Contributed Talk Abstracts
Yiannis Andrianakis, National Oceanography Centre, Southampton
Page3 of 9
University of Bern
Clément Chevalier
Contributed Talk
Multipoint sampling criteria for the identification of an excursion set
Reliability studies in engineering often involve the identification of potentially dangerous configurations of a system
which depends on several real-valued input parameters. Mathematically, if our system is modeled with an expensive to
evaluate simulator f providing a scalar response, we often would like to locate the so called “set of failure”, or “excursion
set” which is the set of all configurations x = (x_1,…,x_d) such that the scalar response f(x) is higher than a
predetermined threshold T.
Many strategies relying on a Kriging metamodel have already been proposed to deal with this problem in the case where
f is expensive to evaluate. However, the criteria proposed so far are either suboptimal, or very difficult to use in real life
applications because they turn out to be themselves expensive to evaluate. Also, they are not adapted to the case of
several cpu available in parallel. In this talk we derive parallel Stepwise Uncertainty Reduction strategies. We show that
they can be computed at a very reasonable cost, and we compare their performance to non-parallel strategies. An
application in nuclear safety is presented, as well as further very promising results on multivariate toy functions.
JOINT AUTHORS:
David Ginsbourger (University of Bern), Yann Richet (IRSN)
Durham University
Jonathan Cumming
Contributed Talk
Emulating expensive decision choices with application to computer models of complex physical
systems
Most experiments on physical systems require the specification of various decision parameters. When such systems are
modelled by computer simulators, then we must optimise our choice of decision inputs for the computer simulator while
recognising all of the other sources of uncertainty which are characteristic of problems in computer modelling. One such
example is found in well testing in hydrocarbon reservoirs, where the parameters of the well test (such as its duration in
time) must be chosen before the experiment is performed to learn about the geological configuration of the reservoir.
While the parameters of the well test are controllable, the configuration of the reservoir is unknown and to be
discovered. Given such a problem, we consider the goal of identifying good choices of decision parameters with respect
to the uncertainty in the reservoir's configuration. We approach this problem by using a simple computer model of
reservoir behaviour which can simulate the results of any well test given its geology. We can then assess the utility of a
single choice of decision parameters by simulating many well tests under those decisions and considering differing
geologies. Investigating many different choices of decision parameters allows us to build a picture of the behaviour of
the utility function over the decision space. However, as this function is both uncertain and expensive to evaluate, we
use emulation methods to represent our expected utility surface. Applying history matching techniques to the emulated
expected utility then enables us to identify good regions of the decision space, and hence good test configurations. This
study illustrates a general methodology for practical use of computer simulators to support decision problems of realistic
size and complexity.
JOINT AUTHORS:
UCM 2012 Conference : Contributed Talk Abstracts
Michael Goldstein
Page4 of 9
AIMdyn Inc, USA
Vladimir Fonoberov
Contributed Talk
Global Sensitivity-Based Calibration of Computer Models
The number of model parameters and the number of model outputs increase rapidly with complexity of computer
models. A large number of model parameters and outputs makes it challenging to match static or time-evolving model
outputs to known data. Such model calibration or parameterization requires building a cost function which represents
the difference between model outputs and real data. The cost function is then minimized over the space of model
parameters. When the number of model parameters is large, the optimization time becomes prohibitive. We developed
[1] an efficient algorithm to compute global sensitivity of model outputs with respect to thousands of parameters. The
knowledge of global sensitivities of a computer model, allows us to achieve significant speedup of the model calibration.
We will demonstrate an application of our method to calibrate a real-life model of a complex dynamical system with
about 20 states and 100 parameters. The developed algorithms are implemented in our software GoSUM [2] (Global
Optimizations, Sensitivity and Uncertainty in Models).[1] Vladimir Fonoberov and Igor Mezic, Global Sensitivity and
Reduction of Very High-Dimensional Models with Correlated Inputs, 2012 SIAM Conference on Uncertainty
Quantification[2] Aimdyn GoSUM software, available at http://aimdyn.com
JOINT AUTHORS:
Igor Mezic, University of California, Santa Barbara, USA
University of Exeter
Tom Fricker
Contributed Talk
Evaluating climate predictions: when is past performance a guide to future performance?
This talk is about evaluating the quality of model predictions of complex dynamic systems using past observations. We
frame our ideas using meteorological examples, but the results have implications for evaluating models of any timeevolving system.
Short-term weather forecasting systems are routinely evaluated using historical data. Typically the quality of past
forecasts is measured using a scoring rule, and the average score is used as an estimate of the quality of the next
forecast. Climate scientists attempt to use a similar process for evaluating long-term climate forecasting systems.
Sequences of genuine forecasts of past climate are usually not available, so instead they produce hindcast (retrospective
forecast) sets. Experience shows that the evaluation process for weather forecasting systems produces adequate
estimates of the quality of tomorrow's forecast, but there is evidence that this is not the case for climate forecasts.
In this talk we will explore the statistical problem of predicting the quality of forecasts. By considering why this task is
apparently easy for weather forecasts but difficult for climate forecasts, we identify the assumptions one has to make in
order to infer forecast quality from hindcast quality, and demonstrate the effects of violating these assumptions. The
result of this work is to clarify the assumptions that underlie our trust in climate forecasts, to characterise conditions
under which hindcast performance can be used as a guide to forecast performance, and to provide guidance about how
hindcast experiments should be designed in order to achieve such conditions.
JOINT AUTHORS:
UCM 2012 Conference : Contributed Talk Abstracts
Dr Chris Ferro
Page5 of 9
University of Leeds
John Paul Gosling
Contributed Talk
Multilevel modelling of an oil reservoir simulator: probabilistic Bayesian and Bayes linear
approaches to history matching
Oil reservoir simulators are complex computer-based models that are designed to capture the dynamics of subterranean
oil traps under exploitation. They are used to forecast flows at wells that are used in the production process and to
estimate the amount of resource remaining in a given reservoir. In a recent case study, we focussed on the calibration of
a simulator for a particular reservoir that was active between 1983 and 1999 and for which we have comprehensive sets
of well-output and forcing time series.
Oil reservoir simulators are particularly interesting from a DACE perspective due to the high dimensionality of their
inputs and outputs (potentially, thousands of both depending on the resolution of the model and the simulation length).
In our case study, we performed dimension reduction on the output parameters using a principal variable approach,
which aims to choose a subset of model outputs that are most representative of the complete set of time- and locationindexed outputs. An advantage of the principal variable approach is that the reduced set of variables is simply a subset of
the raw output variables rather than being a set of transformed variables (like the linear combinations from a principal
component analysis). In our case study, we also considered the use of a principal component analysis in the dimension
reduction to highlight the difference between the two techniques.
The reservoir simulator we considered was computationally expensive to run (a single run at the finest resolution took
between five and six hours to run), and we required an emulator to perform our analyses. The simulator could also be
run at lower levels of numerical precision thus saving computing time at the cost of accuracy (the coarsest version could
be run in about five minutes). In this case study, we investigated if we could exploit these computationally cheaper,
coarser simulator runs in the calibration of the model given historical time-series data from the simulated oil-reservoir.
As far as we are aware, our case study involved the first direct comparison of Bayes linear and probabilistic Bayesian
methodologies in a computer experiments setting. Different strategies were used both in building the emulators and in
identifying input regions for 'good' matches. Qualitatively, we found little difference in results: the regions identified
with each approach being fairly similar. In the probabilistic analysis, we carried out the history matching process via
calibration: we identified 'good' input regions by considering the posterior distribution of the calibration inputs. The
Bayes linear approach performed history matching directly, and was computationally less intensive, though not as
informative about the likely location of a 'best input'. This is because it uses a measure of implausibility to rule out parts
of the input space where poor matches occur instead of assigning probabilities to all possible input configurations. We
use this case study to highlight the key differences in the approaches and to explain why those differences occur.
Jeremy Oakley (Sheffield), Jonathan Cumming & Allan Seheult (Durham)
JOINT AUTHORS:
University of Michigan
James Holloway
Contributed Talk
Assessing Uncertainties in Radiative Shock Modeling
The CRASH code is a radiation-hydrodynamics code capable of modeling 1D, 2D and 3D radiative shock experiments. We
are using this code to understand the uncertainties in predictions of a set of radiative shock simulations and
experiments. In our system a 1 ns, 360 J laser pulse ablates a Be disk which drives a shock moving ~100km/s through a
Xe gas. This shock heats the Xe sufficiently that a radiative cooling layer forms, and this radiation pre-heats the Xe ahead
of the shock. The radiation also ablates the wall of the polyimide shock tube, driving another shock radially inwards. The
physics is dominated by the interaction among the primary radiative shock, the ablation-driven wall shock, and the
material interfaces between Xe, Be and polyimide. This physics is relevant to astrophysics and fundamental high-energydensity physics research. Through several series of 1D and 2D computer runs and experimental campaigns we have
collected data on various quantities of interest including shock location, wall shock location, and measures of the radial
distribution of dense Xe. The different simulations have some physics parameters in common and some physics
parameters that exist only in 1D or 2D. We have developed methodologies that can jointly calibrate these different
fidelity of simulators. This requires two discrepancies: one the discrepancy between the 1D and 2D simulations, and one
the discrepancy between the 2D simulation and field measurements. The discrepancy between simulation models
depends on those physics parameters that are unique to the 2D model and the common physics parameters, but not on
those that are unique to the 1D model. We have also explored the process of extrapolating models into new regions of
input space where there are simulation runs but no field measurements. A major benefit of the work has actually been
to stress both the CRASH code and the field experiments in ways that have led to better understanding of uncertainties
in measurements and improvements in the simulation capability.
This research was supported by the DOE NNSA/ASC under the Predictive Science Academic Alliance Program by grant number DEFC5208NA28616.
JOINT AUTHORS:
UCM 2012 Conference : Contributed Talk Abstracts
Joslin Goh, Mike Grosskopf, Bruce Fryxell & Derek Bingham
Page6 of 9
University of Leeds
Lindsay Lee
Contributed Talk
Sensitivity analysis of a complex global aerosol model to quantify parametric uncertainty
Global aerosol contributions to radiative forcing (and hence climate change) are persistently subject to large uncertainty
in successive Intergovernmental Panel on Climate Change (IPCC) reports (Schimel et al., 1996; Penner et al., 2001;
Forster et al., 2007). As such more complex models are being developed to simulate aerosol microphysics in the
atmosphere. The uncertainty in global aerosol model estimates is currently estimated by measuring the diversity
amongst different models (Textor et al., 2006, 2007; Meehl et al., 2007). The uncertainty at the process level due to the
need to parameterise in such models is not yet understood and it is difficult to know whether the added model
complexity comes at a cost of high model uncertainty. In this work the model uncertainty and its sources due to the
uncertain parameters is quantified using variance-based sensitivity analysis.
Due to the complexity of a global aerosol model we use Gaussian process emulation with a sufficient experimental
design to make such as a sensitivity analysis possible. The global aerosol model used here is GLOMAP (Mann et al., 2010)
and we quantify the sensitivity of numerous model outputs to 27 expertly elicited uncertain model parameters
describing emissions and processes such as growth and removal of aerosol. Using the R package DiceKriging (Roustant et
al., 2010) along with the package sensitivity (Pujol, 2008) it has been possible to produce monthly global maps of model
sensitivity to the uncertain parameters over the year 2008.
Global model outputs estimated by the emulator are shown to be consistent with previously published estimates
(Spracklen et al. 2010, Mann et al. 2010) but now we have an associated measure of parameter uncertainty and its
sources. It can be seen that globally some parameters have no effect on the model predictions and any further effort in
their development may be unnecessary, although a structural error in the model might also be responsible. The
sensitive parameters are shown to change regionally due to the local aerosol properties. The quantification of the
relative sensitivities allows modellers to prioritise future model development in order to reduce prediction uncertainty.
For example, model predictions would be improved over polluted regions if the size and rate of particulate emissions
could be better constrained. The result is a complete understanding of the model behaviour, the effect of increasing the
model complexity and the uncertainty introduced.
Professor Ken Carslaw, Dr Kirsty Pringle
JOINT AUTHORS:
CNRS/Univ. Paris-Sud, Orsay, FRANCE
Pascal Pernot
Contributed Talk
Managing uncertain branching ratios in chemical models
Uncertainty and sensitivity analysis are becoming essential tools in the modelling of complex chemical systems. In
domains such as atmospheric, combustion, plasma, astro- or bio- chemistry, modellers use large databases of (uncertain)
reaction rate constants to parametrize the differential equations describing the kinetics of the chemical system and try
to identify "key reactions" explaining a large portion of the model predictions uncertainty.
The present paradigm in those databases is a "one constant per reaction" representation which combines
heterogeneous data and ignores the correlation structures arising from the measurement techniques of these data. In
particular, nothing is done for variables with a sum-to-one constraint, such as the branching ratios of chemical reactions
with multiple product pathways.
We have shown recently that this constraint has to be implemented with care: neglecting the sum-to-one correlation
between branching ratios of a reaction leads to an overestimation of the variance of the reaction products
concentrations, and to a spurious transfer of variance to species not directly involved in the reaction [1].
We addressed this issue by developing a toolbox of knowledge-adapted Dirichlet-type distributions [1-3]. In many
instances, the sets of experimental data about branching ratios are incomplete, with complete indetermination within
subsets of products. Such cases can be handled by Nested Dirichlet distributions (probabilistic trees) [2-3], the interest of
which is twofold: (1) to preserve the correlation structure of experimental data by imposing sum-to-one representations;
and (2) to be able to introduce unknown subsets of branching ratios (missing data) in the models. This approach enables
us to exploit hundreds of experimental data about branching ratios of electron recombination process in chemical
plasmas that were previously ignored by modellers lacking a suitable representation [2-4].
References
1. Carrasco, N. and Pernot, P. (2007). Modeling of branching ratio uncertainty in chemical networks by Dirichlet distributions., J. Phys. Chem. A 111:3507-3512.
2. Plessis, S., Carrasco, N., and Pernot, P. (2010). Knowledge-based probabilistic representations of branching ratios in chemical networks: the case of dissociative
recombination. J. Chem. Phys. 133:134110.
3. Pernot, P., Plessis, S. and Carrasco, N. (2011). Probabilistic representations of partial branching ratios: bridging the gap between experiments and chemical models. J.
Phys.: Conf. Ser. 300:012027.
4. Plessis, S., Carrasco, N., Dobrijevic, M. and Pernot, P. (2012). Production of neutral species in Titan's ionosphere through dissociative recombination of ions as measured
by INMS. Icarus, in press.
JOINT AUTHORS:
UCM 2012 Conference : Contributed Talk Abstracts
S. Plessis, Dr. & N. Carrasco, Dr.
Page7 of 9
University of Sheffield
Mark Strong
Contributed Talk
Computing the expected value of model improvement in the context of health economic decision
making. A model discrepancy based approach
Health economic models predict the costs and health effects associated with competing decision options (e.g.
recommend drug X versus Y). Such models are typically deterministic and `law-driven', rather than fitted to data. Current
practice is to quantify input uncertainty, but to ignore uncertainty due to deficiencies in model structure.
However, ignoring `structural' uncertainty makes it difficult to answer the question: given a relatively simple but
imperfect model, is there value in incorporating additional complexity to better describe the decision problem, or is the
simple model `good enough'?
To address this problem we propose a model discrepancy based approach. Firstly, the model is decomposed into a series
of sub-functions. The decomposition is chosen such that the output of each sub-function is a real world observable
quantity. Next, where it is judged that a sub-function would not necessarily result in the `true' value of the corresponding
real world quantity, even if its inputs were `correct', a discrepancy term is introduced. Beliefs about the discrepancies are
specified via a joint distribution over discrepancies and model inputs.
To answer the question `is the model good enough' we then compute the partial expected value of perfect information
(EVPI) for the discrepancy terms, interpreting this as an upper bound on the `expected value of model improvement'
(EVMI). If the expected value of model improvement is small then we have some reassurance that the model is good
enough for the decision.
We illustrate the approach with a health economic model case study, and include a brief discussion of an efficient
method for computing partial EVPI where model parameters are correlated.
Jeremy Oakley
JOINT AUTHORS:
University of Helsinki
Jarno Vanhatalo
Contributed Talk
Probabilistic calibration of 3D-ecosystem model in Gulf of Finland
Gulf of Finland (GoF) is the eastern most basin of the Baltic Sea. Today, the GoF is suffering from severe eutrophication
which has had drastic consequences to the ecosystem. Improving the environmental status of the Baltic Sea and the GoF
requires large scale reductions from current nitrate and phosphorus emissions. These can be realized through, for
example, land use regulations or building water purification plants.
In order to provide quantitative tools to support the decision making between different management actions, the effects
of alternative water purification actions are studied with ecosystem models that simulate the nutrient flux and algae
concentrations as functions of nitrate and phosphorus loadings to the sea. Such models are deterministic and their
predictive accuracy typically fluctuates during the simulation leading to prediction errors and biases. However, when
making management decisions the uncertainty in the model prediction should be taken explicitly into account.
In this work we use the EIA-SYKE 3D ecosystem model (Kiirikki et al., 2001) to predict the nutrient and chlorophyll-a
concentrations in the near surface water column in GoF. The ecosystem model predictions are calibrated with a
Gaussian process following (Berrocal et al.,2009). The objectives are 1) to detect the areas where the ecosystem model
works reliably and where it gives biased predictions, and 2) to study three management actions from the perspective of
EU's Water Framework Directive (WFD). WFD sets target values for nitrogen,phosphorus and chlorophyll-a
concentrations in the coastal line of Baltic Sea. We use the calibrated ecosystem model to evaluate the probability to
achieve the WFD targets with a given action.
We use a five year time series of measurements to train our model and forecast the effects of the management actions
for the next five years. The results show that the probabilistic calibration improves the predictive power of the
ecosystem model. The results show also, for example, that the ecosystem model consistently overestimates the DIP
concentrations in the western GoF and underestimates the DIN concentrations in the eastern GoF during winters. There
are also a timing mismatchs in predicting the chlo-a concentrations during the spring bloom. Even tough the
management actions studied include large scale reductions they have only mild effects and the probability to achieve the
WFD targets is low for most of the studied coastal regions.
References
Berrocal, V., Gelfand, A. E., and Holland, D. M. (2009). A spatio-temporal downscaler for output from numerical models. Journal of Agricultural,
Biological and Environmental Statistics,15(2):176–197.
Kiirikki, M., Inkala, A., Kuosa, H., Pitkänen, H., Kuusisto, M., and Sarkkula, J. (2001). Evaluating the effects of nutrient load reductions on the biomass of
toxic nitrogenfixing cyanobacteria in the Gulf of Finland, Baltic Sea. Boreal Environment Research, 6:131– 146.
JOINT AUTHORS:
UCM 2012 Conference : Contributed Talk Abstracts
Dr. Arto Inkala, Dr. Laura Tuomi, M.sc. Inari Helle and Dr. Heikki Pitkänen
Page8 of 9
Durham University
Ian Vernon
Contributed Talk
Emulation and Efficient History Matching of Stochastic Systems Biology Models
Systems Biology is a rapidly expanding area within the Biological Sciences. Typically it has involved the modelling of large
chemical reaction networks through the use of ODE's. However, for many intracellular networks, especially those
concerning gene transcription, the discrete number of molecules involved and the inherently stochastic behaviour of the
network become important.
These networks can be accurately modelled as stochastic processes, that possess many unknown reaction rate constants
representing all the various reactions involved. Systems Biologists want to learn about these rate parameters by
comparing the model output to observed data on molecule counts.
By treating these networks as stochastic computer models, we have shown how to apply computer model methodology
in order to learn about the reaction rate parameters. We have developed novel Bayes Linear emulators of stochastic
models and shown how they can be used to perform history matching in order to efficiently identify regions of rate
parameter space that are consistent with observed data. The history matching process is a powerful technique and is
either a complete alternative to calibration or a extremely useful precursor to it. It proceeds using an iterative reduction
of the rate or input parameter space using a series of increasingly more accurate emulators. This process raises several
interesting questions in both the deterministic and stochastic model cases which we will discuss, including when is it best
to change to the next iteration or wave, and how do we apportion an initial budget of runs between future waves.
Work done in collaboration with: Michael Goldstein & Junli Liu, Durham Univ.
JOINT AUTHORS:
University of Nottingham
Richard Wilkinson
Contributed Talk
Diagnostic plots for dynamical systems
If we are given a model of an imperfectly observed dynamical system, how should we assess the model’s predictive
performance and diagnose the source of any errors? Traditional measures of performance, such as the mean-square
prediction error, or the Nash-Sutcliffe efficiency, ignore any uncertainty quantification in the predictions, and can give
misleading impressions about the level of performance. Proper scoring rules do assess the uncertainty quantification,
but even if we use a collection of different scores to emphasise different aspects of the forecasts, we are still only
provided with simple numerical summaries from which it can be difficult or impossible to diagnose the source of the
error.
In this talk I’ll review some of the visual diagnostic tools we can use, and introduce a new collection of plots based on
attribute diagrams that can be used to indicate the source and direction of forecast errors made by dynamical systems.
By combining different aspects of the model prediction (k-step ahead forecasts for different k, and the filtering
distributions) with different scores and plots, we can begin to diagnose which aspect of the model is causing the error. I’ll
highlight the difficulty of this task and illustrate the methods using a hydrological model.
Kamonrat Suphawan & Dr Theo Kypraios, University of Nottingham
JOINT AUTHORS:
University of Sheffield
Ben Youngman
Contributed Talk
Bayesian calibration of multivariate stochastic computer models
This work will introduce a generic approach to calibration of complex, stochastic computer models that is applicable to a
wide range of models. The basis for our method is to emulate the likelihood of model output given inputs and use the
resulting emulator for calibration, ie. to identify regions of input space giving output deemed compatible with (some
measure of) reality. The performance of this approach relies heavily on the emulator's accuracy. We introduce a variety
of methods that improve this accuracy, including a sequential and informed approach to choice of input region on which
the emulator is built, and by accurate quantification of any discrepancy between model and reality. In particular this
approach provides a method for simultaneous calibration of models with multivariate output of potentially different
type. We illustrate this by presenting calibration of a healthcare model that represents UK cancer incidence.
JOINT AUTHORS:
UCM 2012 Conference : Contributed Talk Abstracts
Dr Jeremy Oakley
Page9 of 9