Building an Emulator
Outline
• Recipe for building an emulator – MUCM toolkit
• Screening – which simulator inputs matter
• Design – where to run the simulator
• Model structure – mean and covariance functions
• Estimation / inference – building the emulator
• Validation – making sure the emulator is OK
• Possible extensions
• multiple outputs
• dynamic simulators
• Bayes linear methods
• Summary of emulation
EGU short course – session 2
2
The ‘standard’ problem
The MUCM toolkit recipe
www.mucm.ac.uk
Step 0: Know your simulator
• Before attempting to create an emulator it is
important you understand your simulator
• what are the plausible input ranges
• what constraints are there in input combinations
• what is the output behaviour like
• Ideally you may wish to elicit beliefs about the
distributions of the inputs if these are not known
• at least ranges are needed for all inputs
Step 1: Screening – active inputs
• All serious simulators require more than one input
• the norm is anything from a few to thousands
• all of the basic emulation theory in the toolkit assumes multiple
inputs
• Large numbers of inputs pose computational problems
• dimension reduction techniques have been developed
• output typically depends principally on a few inputs
• Screening seeks to identify the most important inputs for a
given output
• most often the Morris method is used, which is a cheap sensitivity
analysis approximation
EGU short course – session 2
5
Screening: the Morris method
• Basic idea: develop a design that changes one input
at a time, while filling space
which change only one input each step
• Compute:
non-linear effect
• average of the
non-linear effect
• Morris designs based on a series of repeated trajectories
elementary effects (μ)
• variance of the
elementary effects (σ)
no effect
EGU short course – session 2
linear effect
6
Step 2: Design
• Design is all about choosing where to run the
simulator to learn a good emulator
• we also need to consider validation and calibration
• There are many options for design and many issues
• in the absence of additional information space-filling
designs are used
• grids are infeasible for all but trivial simulators
Training sample design
• To build an emulator, we use a set of simulator runs
• our training data are y1 = f(x1), ..., yn = f(xn)
• x1, x2, ..., xn are n different points in the space of inputs
• this set of n points is a design
• A good design will provide us with maximum information
about the simulator
• and hence an emulator that is as good as possible
EGU short course – session 2
8
Latin hypercube designs
• LHC designs
• use n values for each input
• combining randomly
• Advantages
• doesn’t necessarily require
a large number of points
• nothing lost if some inputs
are inactive
• Disadvantages
• random choice may not produce an even spread of points
• need to generate many LHC designs and pick the best
EGU short course – session 2
9
EGU short course – session 2
10
Some more design choices
• Various formulae and algorithms exist to generate spacefilling designs for any number of inputs
• the Sobol sequence is often used
• quick and convenient
• not always good when some inputs are inactive
• Optimal designs maximise/minimise some criterion
• e.g. maximum entropy designs
• can be hard to compute, often not massive gains
• Hybrid designs try to satisfy two criteria
• space-filling but also having a few points closer together
• in order to estimate correlation lengths well
EGU short course – session 2
11
Step 3: Building the emulator
• In deciding on the structure of the emulator we have
many choices to make:
• the mean function
• the covariance function
• the prior specifications
• There are no universal solutions here, so judgement
and validation play an important role
EGU short course – session 2
12
The technical part (overview!)
• The emulator is a Gaussian process with:
• mean function m(x) = h(x)Tβ, with h(x) typically [1,x]
• covariance function σ2c(x,x’) = σ2exp-{(x-x’)TC(x-x’)}
•
C is a diagonal matrix of inverse length scales 1/ δ2
• thus the conditional distribution of the simulator output,
y, given the input, x and parameters (β, σ2, δ) is
multivariate normal
• The choices we make can be important
EGU short course – session 2
13
The GP mean function
• We can use this to say what kind of shape we would expect
•
the output to take as a function of the inputs
Most simulator outputs exhibit some overall trend in
response to varying a single input
• so we usually specify a linear mean function
• slopes (positive or negative) are estimated from the training
data
• the emulator mean smoothes the residuals after fitting the
linear terms
• We can generalise to other kinds of mean function if we have
a clear idea of how the simulator will behave
• the better the mean function the less the GP has to do
EGU short course – session 2
14
Example
• Simulator is
•
•
•
solid line
Dashed line is
linear fit
Blue arrows
indicate fitted
residuals
Without the
linear mean
function, we’d have a horizontal (constant) fit
• and larger residuals
• leading to larger emulator uncertainty
EGU short course – session 2
15
EGU short course – session 2
16
The GP covariance function
• The covariance function determines how ‘wiggly’ the
•
response is to each input
There’s a lot of flexibility here, but standard covariance
functions have a parameter for each input
• these ‘correlation length’ parameters are also estimated from
the training data
• but some care is needed
• For predicting output at untried x, correlation lengths are
important
• they determine how much information comes from nearby
training points
• and hence the emulator accuracy
Prior distributions
• Prior information enters through the form of the mean
function
• and to a lesser extent the covariance function
• But we can also supply prior information through the prior
distributions
• for slope/regression parameters and correlation lengths
• also the overall variance parameter
• Putting in genuine prior information here generally improves
emulator performance
• compared with standard ‘non-informative’ priors
 ( )  1
• e.g.
 ( )  1
 ( 2 )   2
Step 4: Learning the emulator
• We normally proceed using Bayesian inference
• just how Bayesian depends on size of problem
• ideally we would ‘integrate out’ all unknown parameters, but this can
be difficult, requiring MCMC
• Details are on the toolkit, but in summary
• typically one can integrate out the regression coefficients (β) and
variance parameters(σ2)
• optimise (maximum likelihood, or MAP) the covariance length scales
(δ)
• Ignoring uncertainty in length scales can be a problem if they
are not well identified, but typically the mean function does
most of the work
EGU short course – session 2
19
EGU short course – session 2
20
Prediction with the emulator
• Once the (hyper)-parameters of the emulator have
been learnt (or integrated out) one can use the
emulator to predict at a new input what the
simulator output would have been
• this is always a predictive distribution
EGU short course – session 2
21
EGU short course – session 2
22
Step 5: Validating the emulator
• Validating the emulator is
essential
• full probabilistic assessment of
fitness for purpose
• first examine the standardised
residuals, with +/- 2 std
intervals
• visual assessment is often very
helpful and provides diagnostic
information
EGU short course – session 2
23
What is validation?
• What does it mean to validate an emulator?
• compare the emulator’s predictions with the simulator output
• make a validation sample of runs at new input configurations
• the emulator mean is the best prediction and is always wrong
• but the emulator predicts uncertainty around that mean
• The emulator is valid if its expressions of uncertainty are
correct
• actual outputs should fall in 95% intervals 95% of the time
• no less and no more than 95% of the time
• standardised residuals should have zero mean and unit variance
See Bastos and O’Hagan preprint on MUCM website
EGU short course – session 2
24
Measures for validation
• The Mahalanobis distance on a test set
• accounts for the predictive covariance on the test set
• follows an F-distribution so we can check the value is close
to the theoretical one for a given test set size
• A useful diagnostic is the pivoted Cholesky
Should
decomposition of the predictive covariance
follow a t
distribution
Suggests non-stationarySuggests
/ poor predictive
poor length
variance
scale / covariance function
Extensions
What types of simulator are amenable to
emulation?
Many outputs
• Most simulators also produce multiple outputs
• for instance, a climate simulator may predict temperature on a
grid, etc.
• Usually, for any given use of the simulator we are interested
in just one output
• so we can just emulate that one, particularly if it is some
combination of the others, e.g. mean global surface
temperature
• But some problems require multi-output emulation
• again, there are dimension reduction techniques
• All described in the MUCM toolkit
EGU short course – session 2
27
Multi-output emulators
• When we need to emulate several simulator outputs, there
are a number of available approaches
• single output GP with added input(s) indexing the outputs
• for temperature outputs on a grid, make grid coordinates 2
additional inputs
• independent GPs
• multivariate GP
• independent GPs for a linear transformation
• e.g. principal components
• possibility for dimension reduction
• These are all documented in the MUCM toolkit
EGU short course – session 2
28
EGU short course – session 2
29
Dynamic emulation
• Many simulators predict a process evolving in time
• at each time-step the simulator updates the system state
• often driven by external forcing variables at each time-step
• climate models are usually dynamic in this sense
• We are interested in emulating the simulator’s time series of
outputs
• the various forms of multi-output emulation can be used
• or a dynamic emulator, emulating the single time-step
• and then iterating the emulator
• Also documented in the MUCM toolkit
EGU short course – session 2
30
EGU short course – session 2
31
Stochastic emulation
• Other simulators produce non-deterministic outputs
• running a stochastic simulator twice with the same input x
produces randomly different outputs
• Different emulation strategies arise depending on what
aspect of the output is of interest
• interest focuses on the mean
• output has added noise
• which we allow for when building the emulator
• interest focuses on risk of exceeding a threshold
• emulate the distribution and derive the risk
• emulate the risk
• This is not yet covered in the MUCM toolkit
EGU short course – session 2
32
Bayes linear methods
• So far assumed a fully Bayesian framework
• But there is an alternative framework – Bayes linear methods
• based only on first and second order moments
• means, variances, covariances
• avoids making assumptions about distributions
• its predictions are also first and second order moments
• means, variances, covariances but no distributions
• The toolkit contains theory and procedures for Bayes linear
emulators
EGU short course – session 2
33
EGU short course – session 2
34
Bayes linear emulators
• Much of the mathematics is very similar
• a Bayes linear emulator is not a GP but gives the same mean
and variance predictions
•
•
for given correlation lengths, mean function parameters
although these are handled differently
• but the emulator predictions no longer have distributions
• Compared with GP emulators
• advantages – simpler and may be feasible for more complex
problems
• disadvantages – absence of distributions limits many of the uses
of emulators
•
compromises made
EGU short course – session 2
35
Summary and Limitations
Why emulation is not always a silver bullet
Some caveats on emulation
• Not all simulators are suitable for emulation
• with very large numbers of (>50) outputs need specific
emulators and large training sets
•
for the problem you are solving are all outputs needed?
• for dynamic simulators with high dimensional state spaces
there remain computational issues
• with discrete inputs and outputs Gaussian processes are
not well suited
• But these issues are being addressed actively in
research projects across the world including MUCM
EGU short course – session 2
37
Typical sequence of emulation
Define the problem you
want to solve, identify
the simulator
Identify the inputs,
define ranges and
screen to select
Design the training
set and run the
simulator
Validate the emulator
and if necessary refine
Train the emulator
using the training set
and inference method
Choose the emulator
(mean and covariance)
and define priors
Use the emulator
and if necessary refine
Modify the simulator
or refine it, maybe
using observations
EGU short course – session 2
38
Summary
• Before you emulate know your simulator!
• Think carefully about the problem you really want to
solve
• emulation is a tool to solve interesting problems and not
an aim in itself
• The more prior knowledge you bring the easier the
task will be
• choosing mean and covariance, eliciting priors
• Spend time on validation and refinement
• Building an emulator will help you understand your
simulators … not replace them!
EGU short course – session 2
39