Designing Ensembles
for Climate Prediction
Peter Challenor
National Oceanography Centre
Why Ensembles for
Climate Prediction?
• Not just a point estimate
• Uncertainty estimates as well
• Calibration of models against data
• Sensitivity analysis
Overview
• What is experimental design?
• Why should we be interested?
• Perturbed physics ensembles
• Space filling designs
• Some recent results
• Multimodel ensembles
• Conclusions
What is experimental
design?
• Developed from
agricultural
experiments in the
1920’s
• How should you apply
treatments to
experimental plots in a
field?
R.A. Fisher
• Greatest Statistician of
the
th
20
Century
• Randomisation
• Block designs
• Latin squares
• Split plots …
Clinical Trials
• Randomisation
• Blind and Double Blind
Trials
• Sequential Designs
Why should we worry
about designing our
experiments?
• Would you take medication that hadn’t
been through a properly designed
clinical trial?
• Would you set climate policy without a
properly designed climate model
experiment?
Computer
Experiments
(Climate model
• Computerensembles)
experiments are very
different from either clinical trials or field
experiments.
• In general we are using them to explore
the properties of some computer
simulator (model). This is usually the
numerical solution of a system of PDE’s
or ODE’s
Computer
Experiments
• Mathematically we can write our
computer simulator as
• where
y is the output, x is the input
and η(.) is the unknown mathematical
function represented by the simulator
• y and x are often very high dimension
Computer
Experiments
• Normally the purpose of our computer
experiment is to make some inference
about the model
• Estimate what the model does at
inputs we haven’t run it at
• Optimise the model parameters w.r.to
some data
• Make predictions
Types of ensemble
1. Perturbed physics ensembles
Change inputs (parameters, initial
conditions, …) to a single model
2. Multimodel ensembles
Look at multiple models
What is a good
experimental design?
Make our inferences to the highest
accuracy with the minimum cost
Optimal Design
• Fisher Information ≃ inverse of variance
• Maximise the information = minimising
the variance
• D-optimal designs
• Minimise the determinant of the
variance matrix
• A-optimal designs
• Minimise the trace of the variance
matrix
Bayesian Design
• Set up a D-optimal design by
maximising the utility
is the variance matrix of θ in design
S
An Possible Design
• ‘Star’ design
• No aphorism is more frequently
repeated in connection with field trials
experiments, than that we must ask
Nature few questions or, ideally, one
question, at a time. The writer is
convinced that this view is wholly
mistaken. Nature, he suggests, will best
respond to a logical and carefully
thought out questionnaire; indeed, if we
ask her a single question, she will often
refuse to answer until some other topic
has been discussed.
• R.A. Fisher, 1926
What we need from the
design of a climate model
ensemble
We want to
1. Span the whole input space
2. Observe interactions
3. Minimise the number of simulator
evaluations
Space Filling Designs
• Factorial Designs
• Latin Hypercubes
• Pseudo random sequences
• Sobol sequence
The Full Factorial
• We set each input (factor) at a set
number of levels
• All combinations are included in the
design
• n levels of m factors needs n
• This gets large quickly
m
points
An Example
•5
2
factorial
Fractional Factorial
• Full factorials are expensive
• For large number of factors only 2 or 3
levels
• Can use fractional factorials (2 levels)
The Latin Hypercube
• Decide how many simulator runs you
can afford
• Divide each input range into that
number of intervals
• Allocate a point to each interval
• Randomly permute across each input
The Latin Hypercube
The Latin Hypercube
• We don’t have an algorithm for the
optimal Latin hypercube
• What is a good Latin hypercube?
‣
Maximin
‣
Orthogonal designs
‣
Pragmatic designs
A Latin Hypercube
A maximin LHC
Are Factorials better
than Latin
Hypercubes
Low Discrepancy
Sequences
• Alternative to Latin hypercubes
• Designed for multi-dimensional integrals
• Examples include Halton sequences,
Niederreiter nets and Sobol sequences
Sobol Sequences
• A low discrepancy sequence
• A 2 -1 Sobol sequence is a Latin
n
hypercube
• Some projections of multi-dimensional
Sobol sequences are not ‘good’
Sobol Sequences
Sobol Sequences
Sequential Designs
• So far our designs have been one off
• We make a design and that dictates
how we run the simulator
• We do not learn from the early runs
• An idea from clinical trials is to learn as
we carry out the experiment
Sequential Design for
Computer
Experiments
• Perform an initial experiment (usually
space filling)
• Add additional points to satisfy some
•
criteria
We might add additional points for
where our
optimisation
predictions of simulator output are most
uncertain
A D-optimal design
for smoothness
• I’m fitting an emulator to a computer
experiment
• Can we design an experiment to
estimate the ‘smoothness’ parameters
of the emulator optimally?
Emulators
• δ is a zero mean Gaussian process
• This is defined in terms of a variance
2
(σ and
a correlation function (C(x1,x2))
An Example of an
Emulator
Zhu and Stein (2004)
• In the geostatistical context Zhu and
Stein show that the Fisher information
is approximately given by
Bayesian Design
• Approximate the inverse of the
covariance matrix by the Fisher
information matrix
• Set up a D-optimal design by
maximising the utility
5-point Sobol
10-point Sobol
Sobol 10 +5
One at time (5)
Five at a time
Designing for Multiple
Climate Models
• So far we have considered designs for
single simulators
• How might we design for multiple
models?
• The IPCC problem
• ‘Ensemble of opportunity’?
So What’s the
Problem
• Common outputs between simulators
• Not common inputs
• An important area for research
Conclusions
• Designing model ensembles can
- make them more efficient
- make the experimenter think about
the problem
• There are a variety of designs around
• Consult a statistician before you design
the experiment
• Design of computer experiments is an
active area of research (not only in
climate/environmental sciences)