UQ: from end to end
Tony O’Hagan
Outline
Session 1: Quantifying input uncertainty
Information
Modelling
Elicitation
Coffee break: Propagation!
Session 2: Model discrepancy and calibration
All models are wrong
Impact of model discrepancy
Modelling model discrepancy
12/8/2014
UQ Summerschool 2014
2
UQ: from end to end
Session 1: Quantifying input
uncertainty
Context
You have a model
To simulate or predict some real-world process
I’ll call it a simulator
For a given use of the simulator you are unsure of the
true or correct values of inputs
This uncertainty is a major component of UQ
Propagating it through the simulator is a fundamental step in UQ
We need to express that uncertainty in the form of
probability distributions
But how?
I feel that this is a neglected area in UQ
Distributions assumed, often with no discussion of where they came
from
12/8/2014
UQ Summerschool 2014
4
Focus of this session
Probability distributions for inputs
Representing the analyst’s knowledge/uncertainty
What they mean
Interpretation of probability
Where they come from
Analysis of data and/or judgement
Elicitation
Principles
Single input
Multiple inputs
Multiple experts
12/8/2014
UQ Summerschool 2014
5
The analyst
The distributions should represent the best knowledge of
the model user about the inputs
I will refer to the model user as the analyst
They are the analyst’s responsibility
The analyst is the one who is interested in the simulator output
For a particular application
And some or all of the inputs refer specifically to that application
The analyst must own the input distributions
They should represent best knowledge
Obviously!
Anything else is unscientific
Less input uncertainty means (generally) less output uncertainty
12/8/2014
UQ Summerschool 2014
6
What probability?
Before we go further, we need to understand how a
probability distribution represents someone’s knowledge
The question goes right to the heart of what probability means
Example:
We are interested in X = the proportion of people infected with
HIV who will develop AIDS within 10 years
when treated with a new drug
X will be an input to a clinical trial simulator
To assist the pharmaceutical company in designing the drug’s
development plan
Analyst Mary expresses a probability distribution for X
12/8/2014
UQ Summerschool 2014
7
Mary’s distribution
The stated distribution is shown
on the right
It specifies how probable any
particular values of X are
E.g. It says there is a probability
of almost 0.7 that X is below 0.4
And the expected value of X is
0.35
It even gives a nontrivial
probability to X being less than
0.2
Which would represent a major
reduction in HIV progression
12/8/2014
UQ Summerschool 2014
8
How can X have probabilities?
Almost everyone learning probability is taught the
frequency interpretation
The probability of something is the long run relative frequency
with which it occurs in a very long sequence of repetitions
How can we have repetitions of X?
It’s a one-off: it will only ever have one value
It’s that unique value we’re interested in
Simulator inputs are almost always like this – they’re one-off!
Mary’s distribution can’t be a probability distribution in
that sense
So what do her probabilities actually mean?
And does she know?
12/8/2014
UQ Summerschool 2014
9
Mary’s probabilities
Mary’s probability 0.7 that X < 0.4 is a judgement
She thinks it’s more likely to be below 0.4 than above
So in principle she would bet even money on it
In fact she would bet £2 to win £1 (because 0.7 > 2/3)
Her expectation of 0.35 is a kind of best estimate
Not a long run average over many repetitions
Her probabilities are an expression of her beliefs
They are personal judgements
You or I would have different probabilities
We want her judgements because she’s the expert!
We need a new definition of probability
12/8/2014
UQ Summerschool 2014
10
Subjective probability
The probability of a proposition E is a measure of a
person’s degree of belief in the truth of E
If they are certain that E is true then P(E) = 1
If they are certain it is false then P(E) = 0
Otherwise P(E) lies between these two extremes
Exercise 1 – How many Muslims in Britain?
Refer to the two questions on your sheet
The first asks for a probability
Make your own personal judgement
If you don’t already have a good feel for the probability scale, you
may find it useful to think about betting odds
The second asks for another probability
12/8/2014
UQ Summerschool 2014
11
Subjective includes frequency
The frequency and subjective definitions of probability
are compatible
If the results of a very long sequence of repetitions are
available, they agree
Frequency probability equates to the long run frequency
All observers who accept the sequence as comprising repetitions
will give that frequency as their (personal/subjective) probability
for the next (or any future) result in the sequence
Subjective probability extends frequency probability
But also seamlessly covers propositions that are not repeatable
It’s also more controversial
12/8/2014
UQ Summerschool 2014
12
It doesn’t include prejudice etc!
The word “subjective” has derogatory overtones
Subjectivity should not admit prejudice, bias, superstition, wishful
thinking, sloppy thinking, manipulation ...
Subjective probabilities are judgements but they should
be careful, honest, informed judgements
As “objective” as possible without ducking the issue
Using best practice
Formal elicitation methods
Bayesian analysis
Probability judgements go along with all the other
judgements that a scientist necessarily makes
And should be argued for in the same careful, honest and
informed way
12/8/2014
UQ Summerschool 2014
13
What about data?
I’ve presented the analyst’s probability distributions as a
matter of pure subjective judgement – what about data?
Many possible scenarios:
X is a parameter for which there is a published value
Analyst has one or more direct experimental evaluations for X
Analyst has data relating more or less directly to X
Analyst has some hard data but also personal expertise about X
Analyst relies on personal expertise about X
Analyst seeks input from an expert on X
…
12/8/2014
UQ Summerschool 2014
14
The case of a published value
The published value may come with a completely
characterised probability distribution for X representing
uncertainty in the value
The analyst simply accepts this distribution as her own
judgement
Or it may not
The analyst needs to consider the uncertainty in X around the
published value P
X = P + E, where E is the error
Analyst formulates her own probability distribution for E
The published value P may simply come with a standard
deviation
The analyst accepts this as one judgement about E
12/8/2014
UQ Summerschool 2014
15
Using data – principles
The appropriate framework for using data is Bayesian
statistics
Because it delivers a probability distribution for X
Classical frequentist statistics can’t do that
Even a confidence interval is not a probability statement about X
The data are related to X through a likelihood function
Derived from a statistical model
This is combined with whatever additional knowledge the
analyst may have
In the form of a prior distribution
Combination is performed by Bayes’ theorem
The result is the analyst’s posterior distribution for X
12/8/2014
UQ Summerschool 2014
16
Using data – practicalities
If data are highly informative about X, prior information
may not matter
Use a conventional non-informative prior distribution
Otherwise the analyst formulates her own prior distribution for X
Bayesian analysis can be complex
Analyst is likely to need the services of a Bayesian statistician
The likelihood/model is also a matter of judgement!
Although I will not delve into this today
12/8/2014
UQ Summerschool 2014
17
Summary
We have identified several situations where distributions
need to be formulated by personal judgement
No good data – analyst formulates distribution for X
Published data does not have complete characterisation of
uncertainty – analyst formulates distribution for E
Data supplemented by additional expertise – analyst formulates
prior distribution for X
Analyst may seek judgements of one or more experts
Rather than relying on her own
Particularly when the stakes are high
We have identified just one situation where personal
judgements are not needed
Published data with completely characterised uncertainty
12/8/2014
UQ Summerschool 2014
18
Elicitation
The process of
representing the knowledge
of one or more persons (experts)
concerning an uncertain quantity
as a probability distribution for that quantity
Typically conducted as a dialogue between
the experts – who have substantive knowledge about the
quantity (or quantities) of interest – and
a facilitator – who has expertise in the process of elicitation
ideally face to face
but may also be done by video-conference, teleconference or online
12/8/2014
UQ Summerschool 2014
19
Some history
The idea of formally representing uncertainty using
subjective probability judgements began to be taken
seriously in the 1960s
For instance, for judgement of extreme risks
Psychologists became interested
How do people make probability judgements?
What mental processes are used, and what does this tell us
about the brain’s processing generally?
They found many ways that we make bad judgements
The heuristics and biases movement
And continued to look mostly at how we get it wrong
Since this told them a lot about our mental processes
12/8/2014
UQ Summerschool 2014
20
Meanwhile ...
Statisticians increasingly made use of subjective
probabilities
Growth of Bayesian statistics
Some formal elicitation but mostly unstructured judgements
Little awareness of the work in psychology
Reinforced recently by UQ with uncertain simulator inputs
Our interests are more complex
Not really interested in single probabilities
Whole probability distributions
Multivariate distributions
We want to know how to get it right
Psychology provides almost no help with these challenges
12/8/2014
UQ Summerschool 2014
21
Heuristics and biases
Our brains evolved to make quick decisions
Heuristics are short-cut reasoning techniques
Allow us to make good judgements quickly in familiar situations
Judgement of probability is not something that we
evolved to do well
The old heuristics now produce biases
Anchoring and adjustment
Availability
Overconfidence
And many others
12/8/2014
UQ Summerschool 2014
22
Anchoring and adjustment
Exercise 1 was designed to exhibit this heuristic
The probabilities should on average be different in the two groups
When asked to make two related judgements, the second is
affected by the first
The second is judged relative to the first
By adjustment away from the first judgement
The first is called the anchor
Adjustment is typically inadequate
Second response too close to the first (anchor)
Anchoring can be strong even when obviously
not really relevant to the second question
Just putting any numbers into the discussion
creates anchors
Exercise 1
12/8/2014
UQ Summerschool 2014
23
Availability
The probability of an event is judged more likely if we can quickly
bring to mind instances of it
Things that are more memorable are deemed more probable
High profile train accidents in the UK lead people to imagine rail
travel is more risky than it really is
My judgement of the risk of dying from a particular disease will be
increased if I know (of) people who have the disease or have died
from it
Important for analyst to review all the evidence
12/8/2014
UQ Summerschool 2014
24
Overconfidence
It is generally said that experts are overconfident
When asked to give 95% intervals, say, far fewer than 95% contain the
true value
Several possible explanations
Wish to demonstrate expertise
Anchoring to a central estimate
Difficulty of judging extreme events
Not thinking ‘outside the box’
Expertise often consists of specialist heuristics
Situations we elicit judgements on are not typical
Probably over-stated as a general phenomenon
Experts can be under-confident if afraid of consequences
A matter of personality and feeling of security
Evidence of over-confidence is not from real experts making
judgements on serious questions
12/8/2014
UQ Summerschool 2014
25
The keys to good elicitation
First, pay attention to the literature on psychology of
elicitation
How you ask a question influences the answer
Second, ask about the right things
Things that experts are likely to assess most accurately
Third, prepare thoroughly
Provide help and training for experts
These are built into the SHELF system
Sheffield Elicitation Framework
12/8/2014
UQ Summerschool 2014
26
The SHELF system
SHELF is a package of documents and simple software
to aid elicitation
General advice on conducting the elicitation
Templates for recording the elicitation
Suitable for several different basic methods
Annotated versions of the templates with detailed guidance
Some R functions for fitting distributions and providing feedback
SHELF is freely available and comments and
suggestions for additions are welcomed
Developed by Tony O’Hagan and Jeremy Oakley
R functions by Jeremy
http://tonyohagan.co.uk/shelf
12/8/2014
UQ Summerschool 2014
27
A SHELF template
Word document
Facilitator follows a
carefully constructed
sequence of questions
Final step invites
experts to give their
own feed-back
The tertile method
One of several
supported in SHELF
12/8/2014
UQ Summerschool 2014
28
Annotated template
For facilitator’s guidance
Advice on each field
of the template
Ordinary text says
what is required in
each field
Text in brackets gives
advice on how to work
with experts
Text in italics says why
we are doing it this way
Based on findings in
psychology
12/8/2014
UQ Summerschool 2014
Let’s see how it works
SHELF templates provide a carefully structured
sequence of steps
Informed by psychology and practical experience
I’ll work through these, using the following illustrative
example
An Expert is asked for her judgements about the distance D
between the airports of Paris Charles de Gaulle and Chicago
O’Hare
in miles
She has experience of flying distances but has not flown this
route before
She knows that from LHR to JFK is about 3500 miles
12/8/2014
UQ Summerschool 2014
30
Credible range L to U
Expert is asked for lower and upper credible bounds
Expert would be very surprised if X was found to be below the
lower credible bound or above the upper credible bound
It’s not impossible to be outside the credible range, just highly
unlikely
Practical interpretation might be a probability of 1% that X is
below L and 1% that it’s above U
Example
Expert sets lower bound L = 3500
CDG to ORD surely more than LHR to JFK
Upper bound U = 5000
Additional flying distance for CDG to ORD surely less than 1500
12/8/2014
UQ Summerschool 2014
31
The median M
The value of x for which the expert judges X to be
equally likely to be above or below x
Probability 0.5 (or 50%) below
and 0.5 above
Like a toss of a coin
Or chopping the range into two equally
probable parts
If the expert were asked to choose either
to bet on X < x or on X > x, he/she should have no preference
It’s a specific kind of ‘estimate’ of X
Need to think, not just go for
mid-point of the credible range
L = 0, U =
1
M = 0.36
Example
Expert chooses median M = 4000
12/8/2014
UQ Summerschool 2014
32
The quartiles Q1 and Q3
The lower quartile Q1 is the p = 25% quantile
The expert judges X < x to have probability 0.25
Like tossing two successive Heads with a coin
Equivalently, x divides the range below the median into
two equi-probable parts
‘Less than Q1’ & ‘between Q1 and M’
Should generally be closer to M than Q1
L = 0, U = 1
M = 0.36
Q1 = 0.25
Q3 = 0.49
Similarly, upper quartile Q3 is
p = 75%
Q1, M and Q3 divide the range into
four equi-probable parts
Example
Expert chooses Q1 = 3850, Q3 = 4300
12/8/2014
UQ Summerschool 2014
33
Then fit a distribution
Any convenient distribution
As long as it fits the elicited summaries adequately
SHELF has software for fitting a range of standard distributions
At this point, the choice should not matter
The idea is that we have elicited enough
Any reasonable distribution choice will be similar to any other
Elicitation can never be exact
The elicited summaries are only approximate anyway
If the choice does matter
i.e. different fitted distributions give different answers to the problem
for which we are doing the elicitation
We can try to remove the sensitivity by eliciting more summaries
Or involving more experts
12/8/2014
UQ Summerschool 2014
34
Exercise 2
So let’s do it!
We’re going to elicit your beliefs about one of the
following (you can choose!)
Number of gold medals to be won by China in 2016 Olympics
Length of the Yangtze River
Population of Beijing in 2011
Proportion of the total world land area covered by China
12/8/2014
UQ Summerschool 2014
35
Do we need a facilitator?
Yes, if the simulator output is sufficiently important
A skilled facilitator is essential to get the most accurate and
reliable representation of the expert’s knowledge
At least for the most influential inputs
Otherwise, no
The analyst can simply quantify her own judgements
But it’s still very useful to follow the SHELF process
In effect, the analyst interrogates herself
Playing the role of facilitator as well as that of expert
12/8/2014
UQ Summerschool 2014
36
Multiple inputs
Hitherto we’ve basically considered just one input X
In practice, simulators almost always have multiple
inputs
Then we need to think about dependence
Two or more uncertain quantities are independent if:
When you learn something about one of them it doesn’t change
your beliefs about the others
It’s a personal judgement, like everything else in elicitation!
They may be independent for one expert but not for another
Independence is nice
Independent inputs can just be elicited separately
12/8/2014
UQ Summerschool 2014
37
Exercise 3
1.
2.
3.
4.
Which of the following sets of quantities would you
consider independent?
The average weight B of black US males aged 40 and
the average weight W of white US males aged 40
My height H and my age A
The time T taken by the Japanese bullet train to travel
from Tokyo to Kyoto and the distance D travelled
The atomic numbers of Calcium (Ca), Silver (Ag) and
Arsenic (As)
12/8/2014
UQ Summerschool 2014
38
Eliciting dependence
If quantities are not independent we must elicit the
nature and magnitude of dependence between them
Remembering that probabilities are the best summaries
to elicit
Joint probabilities
Probability that X takes some x-values and Y takes some y-values
Conditional probabilities
Probability that Y takes some y-values if X takes some x-values
Much harder to think about than probabilities for a single
quantity
Perhaps the simplest is the quadrant probability
Probability both X and Y are above their individual medians
12/8/2014
UQ Summerschool 2014
39
Bivariate quadrant probability
First elicit medians
Now elicit quadrant probability
?
0.5
Value indicates direction and
strength of dependence
Median
0.5
Median
12/8/2014
It can’t be negative
Or more than 0.5
0.25 if X and Y are independent
Greater if positively correlated
0.5 if when one is above its
median the other must be
Less than 0.25 if negatively
correlated
Zero if they can’t both be above
their medians
UQ Summerschool 2014
40
Higher dimensions
This is already hard
Just for two uncertain quantities
In order to elicit dependence in any depth we will need to elicit
several more joint or conditional probabilities
More than two variables – more complex still!
Even with just three quantities...
Three pairwise bivariate distributions
With constraints
The three-way joint distribution is not implied by those, either
We can’t even visualise or draw it!
There is no clear understanding among elicitation
practitioners on how to elicit dependence
12/8/2014
UQ Summerschool 2014
41
Avoiding the problem
It would be so much easier if the quantities we chose to
elicit were independent
i.e. no dependence or correlation between them
Then eliciting a distribution for each quantity would be enough
We wouldn’t need to elicit multivariate summaries
The trick is to ask about the
right quantities
Redefine inputs so they
become independent
This is called elaboration
Or structuring
3
2
1
-3
-2
-1
y z0
x
0
1
2
3
-1
-2
-3
12/8/2014
UQ Summerschool 2014
42
Example – two treatment effects
A clinical trial will compare a new treatment with an
existing treatment
Existing treatment effect A is relatively well known
Expert has low uncertainty
But added uncertainty due to the effects of the sample population
New treatment effect B is more uncertain
Evidence mainly from small-scale Phase III trial
A and B will not be independent
Mainly because of the trial population effect
If A is at the high end of the expert’s distribution, she would
expect B also to be relatively high
Can we break this dependence with elaboration?
12/8/2014
UQ Summerschool 2014
43
Relative effect
In the two treatments example, note that in clinical trials
attention often focuses on the relative effect R = B/A
When effect is bad, like deaths, this is called relative risk
Expert may judge R to be independent of A
Particularly if random trial effect is assumed multiplicative
If additive we might instead consider A independent of D = B – A
But this is unusual
So elicit separate distributions for R and A
The joint distribution of (A, B) is now implicit
Can be derived if needed
But often the motivating task can be rephrased in terms of (A, R)
12/8/2014
UQ Summerschool 2014
44
Trial effect
Instead of simple structuring with the relative risk R, we
can explicitly recognise the cause of the correlation
Let T be the trial effect due to difference between the trial
patients and the wider population
Let E and N be efficacies of existing and new treatments in the
wider population
Then A = E x T and B = N x T
Expert may be comfortable with independence of T, E and N
With E well known, T fairly well known and N more uncertain
We now have to elicit distributions for three quantities
instead of two
But can possibly assume them independent
12/8/2014
UQ Summerschool 2014
45
General principles
Independence or dependence are in the head of the
expert
Two quantities are dependent if learning about one of them
would change his/her beliefs about the other
Explore possible structures with the expert(s)
Find out how they think about these quantities
Expertise often involves learning how to break a problem down
into independent components
SHELF does not yet handle multivariate elicitation
But it does include an explicit structuring step
Which we can now see is potentially very important!
Templates for some special cases expected in the next release
12/8/2014
UQ Summerschool 2014
46
Multiple experts
The case of multiple experts is important
When elicitation is used to provide expert input to a
decision problem with substantial consequences, we
generally want to use the skill of as many experts as
possible
But they will all have different opinions
Different distributions
How do we aggregate them?
In order to get a single elicited distribution
12/8/2014
UQ Summerschool 2014
47
Aggregating expert judgements
1.
Two approaches
Aggregate the distributions
Elicit a distribution from each expert separately
Combine them using a suitable formula
For instance, simply average them
Called ‘mathematical aggregation’ or ‘pooling’
2.
Aggregate the experts
Get the experts together and elicit a single distribution
Called ‘behavioural aggregation’
Neither is without problems
12/8/2014
UQ Summerschool 2014
48
Multiple experts in SHELF
SHELF uses behavioural aggregation
However, distributions are first elicited from experts
separately
After sharing of key information
Allows facilitator to see the range of belief before aggregation
Then experts discuss their differences
With a view to assigning an aggregate distribution
To represent what an impartial, intelligent observer might
reasonably believe after seeing the experts’ judgements and
hearing their discussions
Facilitator can judge whether degree of compromise is
appropriate to the intervening discussion
12/8/2014
UQ Summerschool 2014
49
Challenges in behavioural aggregation
More psychological hazards
Group dynamic – dominant/reticent experts
Tendency to end up more confident
Block votes
Requires careful management
What to do if they can’t agree?
End up with two or more
composite distributions
Need to apply mathematical
pooling to these
But this is rare in practice
12/8/2014
UQ Summerschool 2014
50
Conclusions – Session 1
The analyst needs to supply probability distributions for
uncertain inputs
Probabilities are personal judgements
But as objective and scientific as possible
Distributions should represent her best knowledge
A range of scenarios for specifying distributions
From pure judgement
When no good data are available
To simply using a published value
With quantification of uncertainty around that value
Almost always, some part of the task will require
distributions based on personal judgement
E.g. prior distributions for Bayesian analysis of data
12/8/2014
UQ Summerschool 2014
51
Elicitation is the process of formulating knowledge about
an uncertain quantity as a probability distribution
Many pitfalls, practical and psychological
SHELF is a set of protocols designed to avoid pitfalls
An example of best practice in elicitation
Templates to guide the facilitator or analyst through a structured
sequence of steps
Particular challenges arise when eliciting judgements
about multiple inputs or from multiple experts
12/8/2014
UQ Summerschool 2014
52
And so to the coffee break
Once we have specified input distributions, the next task
is propagation of uncertainty through the simulator
A well studied problem in UQ
Polynomial chaos favoured by engineers, mathematicians
Gaussian process emulators preferred by statisticians
In my opinion, a more powerful and comprehensive UQ approach
12/8/2014
UQ Summerschool 2014
53
UQ: from end to end
Session 2: Model discrepancy and
calibration
Case study – carbon flux
Vegetation can be a major factor in mitigating the
increase of CO2 in the atmosphere
And hence reducing the greenhouse effect
Through photosynthesis, plants take atmospheric CO2
Carbon builds new plant material and O2 is released
But some CO2 is released again
Respiration, death and decay
The net reduction of CO2 is called Net Biosphere
Production (NBP)
I will refer to it as the carbon flux
Complex processes modelled in SDGVM
Sheffield Global Dynamic Vegetation Model
12/8/2014
UQ Summerschool 2014
55
SDGVM C flux outputs for 2000
Map of SDGVM estimates
shows positive flux (C sink)
in North, but negative
(C source) in Midlands
Total estimated flux is
9.06 Mt C
Highly dependent on
weather, so will vary
greatly between years
12/8/2014
UQ Summerschool 2014
56
Quantifying input uncertainties
Plant functional type parameters (growth characteristics)
Expert elicitation
Soil composition (nutrients and decomposition)
Simple analysis from extensive (published) data
Land cover (which PFTs are where)
More complex Bayesian analysis of ‘confusion matrix’ data
12/8/2014
UQ Summerschool 2014
57
Elicitation
Beliefs of expert (developer of SDGVMd) regarding
plausible values of PFT parameters
Four PFTs – Deciduous broadleaf (DBL), evergreen needleleaf
(ENL), crops, grass
Many parameters for each PFT
Key ones identified by preliminary sensitivity analysis
Important to allow for uncertainty about mix of species in a site
and role of parameter in the model
In the case of leaf life span for ENL, this was more
complex
12/8/2014
UQ Summerschool 2014
58
ENL leaf life span
12/8/2014
UQ Summerschool 2014
59
Correlations
PFT parameter value at one site may differ from its value
in another
Because of variation in species mix
Common uncertainty about average over all species induces
correlation
Elicit beliefs about average over whole UK
ENL joint distributions are mixtures of 25 components, with
correlation both between and within years
12/8/2014
UQ Summerschool 2014
60
Soil composition
Percentages of sand, clay and silt, plus bulk density
Soil map available at high resolution
Multiple values in each SDGVM site
Used to form average (central estimate)
And to assess uncertainty (variance)
Augmented to allow for uncertainty in original data (expert
judgement)
Assumed independent between sites
12/8/2014
UQ Summerschool 2014
61
Land cover map
LCM2000 is another high resolution map
Obtained from satellite images
Vegetation in each pixel assigned to one of 26 classes
Aggregated to give proportions of each PFT at each site
But data are uncertain
Field data are available at a
sample of pixels
Countryside Survey 2000
Table of CS2000 class versus
LCM2000 class is called the
confusion matrix
12/8/2014
UQ Summerschool 2014
62
CS2000 versus LCM2000 matrix
LCM2000
DBL
CS2000
ENL Grass Crop
Bare
DBL
66
3
19
4
5
Enl
Grass
Crop
Bare
8
31
7
2
20
5
1
0
1
356
41
3
0
22
289
8
0
15
9
81
Not symmetric
Rather small numbers
Bare is not a PFT and produces zero NBP
12/8/2014
UQ Summerschool 2014
63
Modelling land cover
The matrix tells us about the probability distribution of
LCM2000 class given the true (CS2000) class
Subject to sampling errors
But we need the probability distribution of true PFT given
observed PFT
Posterior probabilities as opposed to likelihoods
We need a prior distribution for land cover
We used observations in a neighbourhood
Implicitly assuming an underlying smooth random field
And the confusion matrix says nothing about spatial
correlation of LCM2000 errors
We again relied on expert judgement
Using a notional equivalent number of independent pixels per site
12/8/2014
UQ Summerschool 2014
64
Overall proportions
Red lines show LCM2000 proportions
Clear overall biases
Analysis gives estimates for all PFTs in each SDGVM site
With variances and correlations
12/8/2014
UQ Summerschool 2014
65
Case study – results
Following on the carbon flux case study, input
uncertainties were propagated through the SDGVM
simulator
Extensive use of Gaussian process emulators
12/8/2014
UQ Summerschool 2014
66
Mean NBP corrections
12/8/2014
UQ Summerschool 2014
67
NBP standard deviations
12/8/2014
UQ Summerschool 2014
68
Aggregate across 4 PFTs
Mean NBP
12/8/2014
Standard deviation
UQ Summerschool 2014
69
England & Wales aggregate
Plug-in estimate
(Mt C)
Mean
(Mt C)
Variance
(Mt C2)
Grass
5.28
4.37
0.2453
Crop
0.85
0.43
0.0327
Deciduous
2.13
1.80
0.0221
Evergreen
0.80
0.86
0.0048
PFT
Covariances
Total
12/8/2014
-0.0081
9.06
7.46
UQ Summerschool 2014
0.2968
70
Sources of uncertainty
The total variance of 0.2968 is made up as follows
Variance due to PFT and soil inputs = 0.2642
Variance due to land cover uncertainty = 0.0105
Variance due to interpolation/emulation = 0.0222
Land cover uncertainty much larger for individual PFT
contributions
Dominates for ENL
But overall tends to cancel out
Changes estimates
Larger mean corrections and smaller overall uncertainty
But we haven’t addressed what is probably the biggest
source of uncertainty in this carbon flux problem …
12/8/2014
UQ Summerschool 2014
71
Notation
A simulator takes a number of inputs and produces a
number of outputs
We can represent any output y as a function
y = f (x)
of a vector x of inputs
12/8/2014
UQ Summerschool 2014
72
Example: A simple machine (SM)
A machine produces an amount of work y which
depends on the amount of effort x put into it
Ideally, y = f(x, β) = βx
Parameter β is the rate at which effort can be converted to work
True value of β is β* = 0.65
Data zi = yi + εi
Graph shows observed data
Points lie below y = 0.65x
For large enough x
Because of losses due to
friction etc.
Large relative to observation
errors
12/8/2014
UQ Summerschool 2014
73
The SM as a simulator
A simulator produces output from inputs
When we consider calibration we divide its inputs into
Calibration parameters – unknown but fixed
Control variables – known features of application context
Calibration concerns learning about the calibration
parameters
Using observations of the real process
Extrapolation concerns predicting the real process
At control variable values beyond where we have observations
We can view the SM as a (very simple) simulator
x is a control variable, β is a calibration parameter
12/8/2014
UQ Summerschool 2014
74
Tuning and physical parameters
Calibration parameters may be physical or just for tuning
We adjust tuning parameters so the model fits reality
better
We are not really interested in their ‘true’ values
We calibrate tuning parameters for prediction
Physical parameters are different
We are often really interested in true physical values
The SM’s efficiency parameter β is physical
It’s the theoretically achievable efficiency in the absence of friction
We like to think that calibration can help us learn about them
12/8/2014
UQ Summerschool 2014
75
Exercise 4
Look at the four datasets, and in each case estimate the
best fitting slope β
Draw a line by eye through the origin
Using a straight-edge
Read off the slope as the y value on the line when x = 1
Write that value beside the graph
The actual best-fitting calibrated values are:
Dataset 1 – 0.58
Dataset 2 – 0.58
Dataset 3 – 0.66
Dataset 4 – 0.57
12/8/2014
UQ Summerschool 2014
76
Calibrating the SM
It’s basically a simple linear regression through the origin
zi = βxi + εi
Calibration
Posterior distribution
misses the true value
completely
More data makes things
worse
More and more tightly
concentrated on the
wrong value
We could use a quadratic
regression but the problem
would remain
12/8/2014
UQ Summerschool 2014
77
The problem is completely general
Calibrating (inverting, tuning, matching) a wrong model
gives parameter estimates that are wrong
Not equal to their true physical values – biased
With more data we become more sure of these wrong values
The SM is a trivial model, but the same conclusions
apply to all models
All models are wrong
In more complex models it is just harder to see what is going
wrong
Even with the SM, it takes a lot of data to see any curvature in
reality
What can we do about this?
12/8/2014
UQ Summerschool 2014
78
Model discrepancy
The SM example suggests that we need to allow that the
model does not correctly represent reality
For any values of the calibration parameters
The simulator outputs deviate systematically from reality
Model discrepancy (or model bias or model error)
There is a difference between the model with best/true
parameter values and reality
r(x) = f(x, θ) + δ(x)
where δ(x) represents this discrepancy
and will typically itself have uncertain parameters
We observe
zi = r(xi) + εi = f(xi, θ) + δ(xi) + εi
12/8/2014
UQ Summerschool 2014
79
SM revisited
Kennedy and O’Hagan (2001) introduced this model
discrepancy
Modelled it as a zero-mean Gaussian process
They claimed it acknowledges additional uncertainty
And mitigates against over-fitting of θ
So add this model discrepancy term to the linear model
of the simple machine
r(x) = βx + δ(x)
With δ(x) modelled as a zero-mean GP
Posterior distribution of β now behaves quite differently
12/8/2014
UQ Summerschool 2014
80
SM – calibration, with discrepancy
Posterior distribution much broader and doesn’t get
worse with more data
But still misses the true value
12/8/2014
UQ Summerschool 2014
81
Interpolation
Main benefit of simple GP model discrepancy is
prediction
E.g. at x = 1.5
Prediction within the range of the data is possible
And gets better with more data
12/8/2014
UQ Summerschool 2014
82
But when it comes to extrapolation …
… at x = 6
More data doesn’t help because it’s all in the range [0, 4]
Prediction OK here but gets worse for larger x
12/8/2014
UQ Summerschool 2014
83
Extrapolation
One reason for wish to learn about physical parameters
Should be better for extrapolation than just tuning
Without model discrepancy
The parameter estimates will be biased
Extrapolation will also be biased
Because best fitting parameter values are different in different parts
of the control variable space
With more data we become more sure of these wrong values
With GP model discrepancy
Extrapolating far from the data does not work
No information about model discrepancy
Prediction just uses the (calibrated) simulator
12/8/2014
UQ Summerschool 2014
84
We haven’t solved the problem
With simple GP model discrepancy the posterior
distribution for θ is typically very wide
Increases the chance that we cover the true value
But is not very helpful
And increasing data does not improve the precision
Similarly, extrapolation with model discrepancy gives
wide prediction intervals
And may still not be wide enough
What’s going wrong here?
12/8/2014
UQ Summerschool 2014
85
Nonidentifiability
Formulation with model discrepancy is not identifiable
For any θ, there is a δ(x) to match reality perfectly
Reality is r(x) = f(x, θ) + δ(x)
Given θ, model discrepancy is δ(x) = r(x) – f(x, θ)
Suppose we had an unlimited number of observations
We would learn reality’s true function r(x) exactly
Within the range of the data
Interpolation works
But we would still not learn θ
It could in principle be anything
And we would still not be able to extrapolate reliably
12/8/2014
UQ Summerschool 2014
86
The joint posterior
Calibration leads to a joint posterior distribution for θ and
δ(x)
But nonidentifiability means there are many equally good
fits (θ, δ(x)) to the data
Induces strong correlation between θ and δ(x)
This may be compounded by the fact that simulators often have
large numbers of parameters
(Near-)redundancy means that different θ values produce (almost)
identical predictions
Sometimes called equifinality
Within this set, the prior distributions for θ and δ(x) count
12/8/2014
UQ Summerschool 2014
87
The importance of prior information
The nonparametric GP term allows the model to fit and
predict reality accurately given enough data
Within the range of the data
But it doesn’t mean physical parameters are correctly
estimated
The separation between original model and discrepancy is
unidentified
Estimates depend on prior information
Unless the real model discrepancy is just the kind expected a
priori the physical parameter estimates will still be biased
To learn about θ in the presence of model discrepancy
we need better prior information
And this is also crucial for extrapolation
12/8/2014
UQ Summerschool 2014
88
Better prior information
For calibration
Prior information about θ and/or δ(x)
We wish to calibrate because prior information about θ is not
strong enough
So prior knowledge of model discrepancy is crucial
In the range of the data
For extrapolation
All this plus good prior knowledge of δ(x) outside the range of the
calibration data
That’s seriously challenging!
In the SM, a model for δ(x) that says it is zero at x = 0, then
increasingly negative, should do better
12/8/2014
UQ Summerschool 2014
89
Inference about the physical parameter
We conditioned
the GP
δ(0) = 0
δ′(0) = 0
δ′(0.5) < 0
δ′(1.5) < 0
12/8/2014
UQ Summerschool 2014
90
Prediction
x = 1.5
12/8/2014
x=6
UQ Summerschool 2014
91
Where is the uncertainty?
Return to the general case
How might the simulator output y = f (x) differ from the
true real-world value z that the simulator is supposed to
predict?
Error in inputs x
Initial values
Forcing inputs
Model parameters
Error in model structure or solution
Wrong, inaccurate or incomplete science
Bugs, solution errors
12/8/2014
UQ Summerschool 2014
92
Quantifying uncertainty
The ideal is to provide a probability distribution p(z) for
the true real-world value
The centre of the distribution is a best estimate
Its spread shows how
much uncertainty about z
is induced by uncertainties
on the previous slide
How do we get this?
Input uncertainty: characterise p(x), propagate through to
p(y)
Model discrepancy: characterise p(z - y)
12/8/2014
UQ Summerschool 2014
93
The hard part
We know pretty well how to do uncertainty propagation
Uncertainties associated with the simulator output
The hard part is the link to reality
The difference between the real system z and the
simulator output y = f(x) using best input values
Because all models are wrong (Box, 1979)
It was through thinking about this link …
Particularly in the context of calibration
i.e. learning about uncertain parameters in the model
And also extrapolation
… that I was led to think more deeply about parameters
And to realise just how important model discrepancy is
12/8/2014
UQ Summerschool 2014
94
Modelling model discrepancy
1.
Three rules:
Must account for model discrepancy
Ignoring it leads to biased calibration, over-optimistic predictions
2.
Discrepancy term must be modelled nonparametrically
Allows learning about reality and interpolative prediction
3.
Model must incorporate realistic knowledge about
discrepancy
To get unbiased learning about physical parameters and
extrapolation
But following these rules is hard
Ongoing research
12/8/2014
UQ Summerschool 2014
95
Managing uncertainty
To understand the implications of different uncertainty
sources
Probabilistic, variance-based sensitivity analysis
Helps with targeting and prioritising research
To reduce uncertainty, get more information!
Informal – more/better science
Tighten p(x) through improved understanding
Tighten p(z - y) through improved modelling or programming
Formal – using real-world data
Calibration – learn about model parameters
Data assimilation – learn about the state variables
Learn about model discrepancy z - y
Validation (another talk!)
12/8/2014
UQ Summerschool 2014
96
Conclusions – Session 2
Without model discrepancy
Inference about physical parameters will be wrong
And will get worse with more data
The same is true of prediction
Both interpolation and extrapolation
With crude GP model discrepancy
Interpolation inference is OK
And gets better with more data
But we still get physical parameters and extrapolation wrong
The better our prior knowledge about model discrepancy
The more chance we have of getting physical parameters right
Also extrapolation
But then we need even better prior knowledge
12/8/2014
UQ Summerschool 2014
97
Any final questions?
It remains just to say thank you for sitting through this
morning’s sessions!
12/8/2014
UQ Summerschool 2014
98