Assessment of uncertainty of
environmental models based on the
equifinality thesis
Keith Beven
Lancaster University, UK
A Manifesto for the Equifinality Thesis
• One aim of environmental models is to achieve a single true
description of governing processes (reality) - especially for
predicting impacts of change
• Difficult to achieve in applications to places that are unique in
their characteristics and where (nonlinear) predictions are
subject to input, observation, and model structural errors
• There may instead be many descriptions that are compatible with
current understanding and available observations
• The concept of the single description may remain a philosophical
axiom or theoretical aim but is impossible to achieve in practice
• So we must accept that there may be many feasible descriptions,
or a concept of equifinality, as the basis for a new approach
A Manifesto for the Equifinality Thesis
So what if we accept multiple possible models and
parameter sets…… <M(θ)1, M(θ)2, … M(θ)N > ???
• There is no optimum model - can only assess the likelihood
of a model being acceptable
• Different acceptable models will produce different
predictions - especially of impacts of future change
• Should therefore assess the resulting uncertainty in
predictions
• Should be prepared to revise predictions as new data
become available
• Models should be rejected if shown to be non-behavioural
(hypothesis testing by model rejection?)
Equifinality: an empirical result
0 .0 0 0 6
0 .0 0 0 6
0 .0 0 0 5
0 .0 0 0 5
0 .0 0 0 4
0 .0 0 0 4
L
L
0 .0 0 0 3
0 .0 0 0 3
0 .0 0 0 2
0 .0 0 0 2
0 .0 0 0 1
0 .0 0 0 1
0
0 .0 2
0 .0 4
0 .0 6
0 .0 8
0 .1
0
0 .0 2
0 .0 4
r
0 .0 6
0 .0 8
0 .1
0 .1 2
0 .1 4
 (c m - 1 )
0 .0 0 0 6
0 .0 0 0 6
0 .0 0 0 5
0 .0 0 0 5
0 .0 0 0 4
0 .0 0 0 4
L
L
0 .0 0 0 3
0 .0 0 0 3
0 .0 0 0 2
0 .0 0 0 2
0 .0 0 0 1
0 .0 0 0 1
1 .5
1 .7
1 .9
2 .1
2 .3
2 .5
0 .1
1
n
10
10 0
K s (c m d ay-1)
Fitting van Genuchten parameters in modelling recharge
after Binley and Beven, Groundwater, 2002
10 0 0
Equifinality: an empirical result
Fitting parameters
of the MAGIC
geochemistry
model
after Page et al.,
Water, Soil and Air
Pollution, 2003
Equifinality: an empirical result
Fitting parameters of the Penman-Monteith equation in
predicting patch scale latent and sensible heat fluxes
after Schulz and Beven, Hydrological Processes, 2003
Uncertainty is not only Statistics
• Classical approach to uncertainty of treating total
model error in linear (or log linear) form
O(X, t) = Ô{M(θ,β)} + ε(X,t)
• Implicit assumption that the model is correct (or can
be made to be correct through another additive
term) and that any error in the inputs and other
boundary conditions cascades through the model
linearly
• Full power of linear statistics can then be used to
estimate
L[O(X,t)|θ]
• and posterior distribution of parameters, θ
Uncertainty is not only Statistics
• But environmental models are generally nonlinear, are
subject to important uncertainties in input and
boundary conditions and (we suspect) suffer from
model structural error
• More interested in L[M(θ)|O(X,t)]
• Additive total error may be difficult to justify but
may be an approximation that will often be useful
• BUT…… from the single total error series we cannot
easily deconstruct the sources of uncertainty in the
modelling process
Sources of subjectivity in assessing model
error
We like to think that we are objective and scientific in modelling but
this is not entirely the case. There are always choices to be made...
• Choice of models to be considered (including processes to be
included, closure and boundary conditions),
• Choice of ranges (prior distributions) of parameter values to be
considered
• Choice of input data (and input data errors) with which to drive
the model (including future scenarios for prediction)
• Choice of error model, performance measure(s) or allowable error
to be considered acceptable (taking account of scale,
heterogeneity and incommensurability effects)
Equifinality and the Modelling Process
• Take a (thoughtful) sample of all possible models
(structures + parameter sets)
• Evaluate those models in terms of both understanding
and observations in a particular application
• Reject those models that are non-behavioural (but note
that there may be a scale problem in comparing model
predictions and observations)
• Devise testable hypotheses to allow further models to be
rejected
• [If all models rejected, revise model structures……]
This is the essence of the GLUE methodology
(Bayesian priors and likelihood functions based
on additive error as special case where strong
assumptions justified)
Published Applications of GLUE
•
•
•
•
•
•
•
•
•
•
•
•
•
Rainfall-runoff modelling
Stochastic rainfall models
Radar Rainfall calibration
Modelling catchment geochemistry
Modelling flood frequency and flood inundation
Modelling river dispersion
Modelling soil erosion
Modelling land surface to atmosphere fluxes
Modelling atmospheric deposition and critical loads
Modelling groundwater capture zones
Modelling groundwater recharge
Modelling water stress cavitation and tree death
Modelling forest fires
Generalised Likelihood Uncertainty
Estimation (GLUE)
Note 1: it is the parameter set in combination with the
given input and boundary condition data that gives a
behavioural model - complex parameter interactions
may mean that marginal distributions and global
covariances have little relevance
Note 2: implicit treatment of complex errors in likelihood
weighting of simulations (effectively assuming that
prediction errors for any behavioural model in
prediction will be “similar” to conditioning periods)
Note 3: prediction limits are conditional on choices
Note 4: depends on having sufficient sample to find
upper limit of performance
Deconstructing total model error
• Extended GLUE methodology
– insist on model providing predictions within range of
“effective observation error” of evaluation variables
– effective observation error constructed to take account of
scale dependencies and incommensurability (may be
dependent on model implementation)
– models providing predictions outside range are rejected as
non-behavioural (multiple models can be included in same
formalism but possible that all models may be rejected)
– success may depend on allowing realisations of error in input
and boundary condition data
Incorporating Observational Errors into GLUE
• Initial condition errors generally poorly known and are
difficult to assess (no means of direct measurement) but
effect will die out as simulation proceeds (eventually)
• Input and boundary condition data errors generally poorly
known because of measurement technique limitations and will
be processed nonlinearly through the model
• Model structural error is ubiquitous and difficult to
assess - claims to physical realism do not exclude possibility
of structural error but want models that are consistent with
effective observational error
• Effective observational error may be difficult to assess
because of scale, heterogeneity and incommensurability
effects - but provides a critical limit for model rejection
Predictive distribution over all
behavioural models: (A) predictions
encompass new observation
1
0.8
0.6
0.4
0.2
0
1.2
1.4
1.6
1.8
2
2.2
Predictive distribution over all
behavioural models: (B) predictions
do not encompass new observation
1
0.8
0.6
0.4
0.2
0
1.2
1.4
1.6
1.8
2
2.2
………But models are still behavioural
(or can be rejected on basis of new observations)
Bukmoongol Catchment – Location
Bukmoongol – Stage/Discharge Errors
0.0
-4.0
Observed Data
Regression Fit
90% Confidence Limits
-6.0
-8.0
log(Discharge) m3/sec
-2.0
-10.0
-7.0
-1.0
1.0
3
Discharge m /sec
Concrete Weir has three
sharp-crested rectangle
sections – 2 only active at
peak discharges
-5.0
-3.0
log(Stage Reading) m
Observed Data
Regression Fit
90% Confidence Limits
0.8
0.6
+/- 20%
0.4
0.2
0.0
0.0
0.1
0.2
0.3
0.4
Stage Reading m
0.5
0.6
Rainfall Magnitude Errors
-5.0
0.5
0.0
0.0
Rainfall Error %
5.0
Event 3
Cumulative P
Cumulative P
Event 2
1.0
1.0
0.5
0.0
-5.0
0.0
Rainfall Error %
5.0
1.0
0.5
0.0
-5.0
0.0
Rainfall Error %
0
0.4
40
Rainfall
Discharge
Cumulative P
Event 1
Discharge
0.2
80
0.0
26/07
31/07
05/08
10/08
15/08
Date Time
20/08
120
25/08
5.0
Results – Model Structural Error?
Standard Z Value
(Scaled PM2)
Log(Discharge)
IC errors
Model
Recession
Error
Timing Errors
And under prediction
Future prospects
•
•
•
•
A possible way forward – To think about moving from evaluations
that are based on ‘lumped’ additive error terms to a framework
that considers error source terms individually
Generic technique, simple to implement in an uncertainty
framework (GLUE) and based on explicit assumptions– even if
assumptions may be imprecise (e.g. “effective measurement
error”)
Allows for equifinality of model structures, input realisations
and parameter sets in producing behavioural models - interaction
of input errors and model structure may be important
Allows for evaluation of model structures by rejection
(hypothesis testing) (see philosophy paper, Proc. Roy. Soc., 2002)
Future prospects: model structural error
•
But what if these tests reveal that all models tried are nonbehavioural and should be rejected?
•
?? Ensure that model space has been searched adequately
•
?? Relax rejection criteria
•
?? Add a model inadequacy term to compensate for structural
error - but will be time variable and will have additional
parameters for each model realisation, and how much correction
should be allowed?
•
?? Find a better model - uncertainty estimation should NOT
remove the need for creative and constructive modelling
and if you might possibly still want to read more…...
•
Beven, K J, 2000, Uniqueness of place and process representations
in hydrological modelling, Hydrology and Earth System Sciences,
4(2), 203-213.
•
Beven, K. J., 2002, Towards an alternative blueprint for a physicallybased digitally simulated hydrologic response modelling system,
Hydrol. Process., 16(2), 189-206
•
Beven, K. J., 2002, Towards a coherent philosophy for environmental
modelling, Proc. Roy. Soc. Lond., A458, 2465-2484 (comment by
Philippe Baveye and reply still to appear)
•
Beven, K J and Young, P C, 2003, Comment on Bayesian Recursive
Parameter Estimation for Hydrologic Models by M Thiemann et al.
Water Resources Research, 39(5), doi: 10.1029/2001WR001183
www.es.lancs.ac.uk/hfdg/glue.html