Error and Uncertainty in
Modeling
George H. Leavesley,
Research Hydrologist,
USGS, Denver, CO
Sources of Error and
Uncertainty
• Model Structure
• Parameters
• Data
• Forecasts of future conditions
Conceptualization of Reality
Sacramento
Effective Parameters and States
real world
heterog.
input
output
measurement
identical
Identical ?
homog.
input
output
(xeff,eff)
model
After Grayson and Blöschl, 2000, Cambridge Univ. Press
Precipitation Measurement
Sparse precip gauge distribution
Mountain blockage of radar
3000M
2000M
1000M
Source: Maddox, et. al. Weather and Forecasting, 2002.
SAHRA – NSF STC
Streamflow Measurement
Accuracy (USGS)
• Excellent
– 95% of daily discharges are within 5% of true value
• Good
– 95% of daily discharges are within 10% of true value
• Fair
– 95% of daily discharges are within 15% of true value
• Poor
– Do not meet Fair criteria
Different accuracies may be attributed to
different parts of a given record
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Performance Measures
Mean (observed vs simulated)
Standard deviation (observed vs simulated)
Root Mean Square
Error (RMSE)
Mean Absolute
Error (MAE)
Performance Measures
Coefficient of Determination R2
Not a good measure -High correlations can be achieved for
mediocre or poor models
Performance Measures
Coefficient of Efficiency E
Widely used in hydrology
Range – infinity to +1.0
Overly sensitive to extreme values
Nash and Sutcliffe, 1970, J. of Hydrology
Seasonal Variability of
Nash-Sutcliffe Efficiency
Seasonal Analysis of Daily Nash-Sucliffe Efficiency
01608500
0.9
0.8
0.7
Efficiency
0.6
0.5
SAC
0.4
GR4J
0.3
PRMS
0.2
0.1
0
-0.1
0
2
4
6
8
Month
10
12
14
Performance Measures
Index of Agreement d
Range 0.0 – 1.0
Overly sensitive to extreme values
Willmott, 1981, Physical Geography
Analysis of Residuals
Performance Measure Issues
• Different measures have different sensitivities
for different parts of the data.
• Are assumptions correct regarding the nature of
the error structures (i.e. zero mean, constant
variance, normality, independence, …)?
• Difficulty in defining what constitutes an
acceptable level of performance for different
types of data.
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Definitions
• Uncertainty analysis: investigation of
the effects of lack of knowledge or
potential errors on model components
and output.
• Sensitivity analysis: the computation of
the effect of changes in input values or
assumptions on model output.
EPA, CREM, 2003
Parameter Sensitivity
The single, “best-fit model”
assumption
Error Propagation
Magnitude of Parameter Error
5%
%> VAR
%> SE
soil_moist_max
10%
20%
50%
0.23963 0.95852 3.83408 23.96303
0.11974 0.47812 1.89901 11.33868
0.15243 0.60973 2.43891 15.24316
%> VAR
%> SE
hamon_coef
0.16210
0.08102
0.10311
0.64840
0.32367
0.41245
2.59359
1.28849
1.64981
16.20993
7.80071
10.31133
%> VAR
%> SE
ssrcoef_sq
0.07889
0.03944
0.05018
0.31556
0.15766
0.20073
1.26224
0.62914
0.80293
7.88900
3.86963
5.01829
joint
0.32477
1.29908
5.19632
32.47698
Objective Function Selection
et
baseflow
soil moisture
routing
gw rech
gw rech
soil moisture
et
routing
Relative Sensitivity
SR = (  QPRED /  PI)
*
(PI / QPRED)
Relative Sensitivity Analysis
Soil available water
holding capacity
Evapotranspiration
coefficient
Relative Sensitivity Analysis
Snow/rain threshold
temperature
Snowfall
adjustment
Parameter Sensitivity
The “parameter equifinality” assumption
• Consider a population of models
• Define the likelihood that they are
consistent with the available data
Regional Sensitivity Analysis
• Apply a random sampling procedure to the
parameter space to create parameter sets
•Classify the resulting model realizations as
“behavioural” (acceptable) or “non-behavioural”
•Significant difference between the set of
“behavioural” and “non-behavioural” parameters
identifies the parameter as sensitive
Spear and Hornberger, 1980, WRR
Regional Sensitivity Analysis
1
F ( i | B )
F ( i )
0
F ( i | B )
i
B = behavioural
Not sensitive
cumulative distribution
cumulative distribution
Sensitive
1
F(  j | B )
F(  j )
F(  j | B )
0
j
B = non-behavioural
Generalized Likelihood Uncertainty
Analysis (GLUE)
• Monte Carlo generated simulations are
classified as behavioural or non-behavioural,
and the latter are rejected.
• The likelihood measures of the behavioural
set are scaled and used to weight the
predictions associated with individual
behavioural parameter sets.
• The modeling uncertainty is then propagated
into the simulation results as confidence
limits of any required percentile.
Dotty Plots and Identifiability Analysis
behavioural
GLUE computed 95%confidence limits
Rockies
Sierras
Objective Function
  obsQ – predQ 
Animas
Cascades
Cle Elum
Carson
300000.0
300000.0
300000.0
250000.0
250000.0
250000.0
200000.0
200000.0
200000.0
150000.0
150000.0
150000.0
100000.0
100000.0
100000.0
50000.0
0.0 0.2 0.4 0.6 0.8
50000.0
0.0 0.2 0.4 0.6 0.8
Parameter
Equifinality
50000.0
0.0 0.2 0.4 0.6 0.8
rad_trncf
300000.0
300000.0
250000.0
250000.0
200000.0
200000.0
150000.0
150000.0
100000.0
100000.0
50000.0
0.0
5.0
10.0
15.0
300000.0
250000.0
Regional
Variability
200000.0
150000.0
100000.0
50000.0
0.0
5.0
10.0
15.0
50000.0
0.0
5.0
10.0
15.0
soil_moist_max (inches)
300000.0
300000.0
300000.0
250000.0
250000.0
250000.0
200000.0
200000.0
200000.0
150000.0
150000.0
150000.0
100000.0
100000.0
100000.0
50000.0
25.0
30.0
35.0
40.0
50000.0
25.0
30.0
35.0
40.0
tmax_allsnow (deg F)
50000.0
25.0
Uncalibrated
Estimate
30.0
35.0
40.0
Multi-criteria Analysis
Increasing the information content of the data
A single objective function:
• cannot capture the many performance attributes
that an experienced hydrologist might look for
• uses only a limited part of the total information
content of a hydrograph
• when used in calibration it will tend to bias model
performance to match a particular aspect of the
hydrograph
A multi-criteria approach overcomes these
problems (Wheater et al., 1986, Gupta et al., 1998,
Boyle et al., 2001, Wagener et al., 2001).
Identifying Characteristic
Behavior
Developing Objective Measures
FD 
1
nD
n Q
obs
Q

Q

2

2
2
peaks/timing
com
D
FQ 
n  Q
n
1
Q
obs
com
quick recession
Q
FS 
1
ns
n Q
obs
s
Q
com
baseflow
Pareto Optimality
F
D
FS
FQ
FS
500 Pareto Solutions
F
D
F
u
n
c
t
i
o
n
S
p
a
c
e
N
o
r
m
a
l
i
z
e
d
P
a
r
a
m
e
t
e
r
S
p
a
c
e
F
Q
SAC-SMA Hydrograph Range
Overall Performance Measures
RMSE min
BIAS min
Parameter Sensitivity by Objective Function
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Evaluation of Model Component
Processes
Solar
Radiation
Observed
SCE
Final
SCE
J
F M A M J
J A
Month
PET
S O
N D
J
Annual
Runoff
1991
1993
F M A M J
J A
Month
1995
Year
1997
1991
1993
1995
Year
Nash-Sutcliffe
1993
N D
Percent Groundwater
Daily Q
1991
S O
1995
Year
1997
Day
1997
Coupling SCA remote
sensing products with point
measures and modeled SWE
to evaluate snow component
process
East River
Integrating
Remotely
Sensed
Data
Percent Basin Snowcover
1
0.8
MODEL
SATELLITE
0.6
0.4
0.2
0
1/0
1/20
2/9
2/29
3/20
4/9
1996
4/29
5/19
6/8
6/28
7/18
Identifiability Analysis
Identification of the model structure and a
corresponding parameter set that are most
representative of the catchment under
investigation, while considering aspects such
as modeling objectives and available data.
Wagener et al., 2001, Hydrology Earth System Sciences
Dynamic Identifiability Analysis - DYNIA
Information content by parameter
Dynamic Identifiability Analysis - DYNIA
Identifiability measure and 90% confidence limits
A Wavelet Analysis Strategy
• Daily time series
Precipitation
• Seasonally varying daily
variance (row sums)
12
100
11
90
10
80
9
70
8
60
• Seasonally varying
variance frequency
decomposition (column
sums)
7
50
6
40
5
30
4
3
20
2
10
1
1
2
3
4
5
6
7
8
• Annual average variance
frequency decomposition
John Schaake, NWS
0
Variance Decomposition
Precipitation
Precipitation
OBSQ
12
100
12
11
90
11
10
80
10
70
9
9
8
6
9
0.45
10
0.4
6
7
6
5
6
4
5
4
20
2
10
1
2
3
4
5
6
7
8
3
3
1
1
2
3
8-day Average Precipitation
20
12
11
18
11
16
10
10
9
4
5
Component
6
7
8
1
8
9
7
5
1
6
4
4
5
6
7
8
0
0.5
0.45
10
0.4
0.35
8
6
5
6
0.3
0.25
4
3
2
2
2
1
0
1
0.2
5
0.15
4
4
3
8
3
11
7
4
7
2
12
6
5
6
0.05
1
7
8
5
2
9
8
Month
10
6
4
0.1
8
12
7
3
3
8-day Avg Variance Transfer Function - Observed
10
9
14
2
0.15
4
OBSQ2
12
1
0.2
2
2
0
0.3
0.25
5
3
0.35
8
7
30
4
0.5
11
7
40
5
12
9
8
Month
50
Variance Transfer Function - Observed
10
8
60
7
1
Variance Transfer
Functions
Streamflow
3
0.1
2
2
0.05
1
1
3
1
2
3
4
5
Component
6
7
8
1
2
3
4
5
6
7
8
0
Streamflow
OBSQ
Estimated Streamflow - Linear
12
10
12
10
11
9
11
9
10
8
10
8
9
7
8
Month
Obs
9
7
8
6
7
6
7
5
5
6
6
4
5
3
4
4
5
3
4
3
2
3
2
2
1
2
1
0
1
10
12
10
9
11
9
8
10
8
1
1
2
3
4
5
Component
6
7
8
1
2
3
ESTQ H
11
10
9
5
6
7
8
9
7
8
0
6
7
5
6
4
5
3
4
7
8
Month
Month
4
ESTQ B
12
PRMS
6
7
5
6
4
5
3
4
3
2
3
2
2
1
2
1
0
1
1
1
2
3
4
5
Component
Linear
6
7
8
1
2
3
4
5
Component
6
7
8
0
SAC
Variance Transfer Functions
OBSH
Variance Transfer Function - Linear
12
0.5
12
0.5
11
0.45
11
0.45
10
0.4
10
0.4
9
0.35
8
Month
Obs
9
0.35
8
0.3
7
0.3
7
0.25
0.25
6
6
0.2
5
0.2
5
0.15
4
0.15
4
3
0.1
3
0.1
2
0.05
2
0.05
0
1
1
1
2
3
4
5
Component
6
7
8
1
2
3
ESTH H
5
6
7
8
0
ESTH B
0.5
12
0.5
11
0.45
11
0.45
0.4
10
0.4
9
9
0.35
8
0.3
7
0.25
6
0.2
5
0.15
4
0.35
8
Month
Month
4
12
10
PRMS
0.3
7
0.25
6
0.2
5
0.15
4
3
0.1
3
0.1
2
0.05
2
0.05
0
1
1
1
2
3
4
5
Component
Linear
6
7
8
1
2
3
4
5
Component
6
7
8
0
SAC
Forecast Uncertainty
Ensemble Streamflow Prediction
Using history as an analog for the future
NOAA
USGS
Probability of
exceedence
BOR
Simulate to today
Predict future using
historic data
2005 ESP
Forecast
ESP Animas River @ Durango
1981
1982
1983
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
12000
8000
6000
4000
2000
9/18/2005
All historic years
ESP - Animas River @ Durango
9/18/2005
9/4/2005
8/21/2005
8/7/2005
7/24/2005
7/10/2005
6/26/2005
6/12/2005
5/29/2005
5/15/2005
5/1/2005
9/4/2005
Forecast Period 4/3 – 9/30
Made 4/2/2005
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
4/3/2005
Observed 2005
Streamflow (cfsd)
Only el nino years
8/21/2005
8/7/2005
7/24/2005
7/10/2005
6/26/2005
6/12/2005
5/29/2005
5/15/2005
5/1/2005
4/17/2005
4/3/2005
0
4/17/2005
Streamflow (cfsd)
10000
1982
1983
1987
1991
1992
1993
1994
1995
1997
1998
2002
2003
2004
2005
Ranked Probability Skill Score (RPSS) for each
forecast day and month using measured runoff and
simulated runoff (Animas River, CO) produced using:
(1) SDS output and (2) ESP technique
RPSS
0.1 0.3
0.5 0.7
0.9
Perfect Forecast: RPSS=1
8
8
6
6
4
4
2
2
0
0
JFMAMJJASOND
JFMAMJJASOND
Month
Month
SDS
ESP
Given current uncertainty in long-term atmosphericmodel forecasts, seasonal to annual forecasts may
be better with ESP
Summary
• This presentation has been a selected review of
uncertainty and error analysis techniques.
• No single approach provides all the information
needed to assess a model. The appropriate mix is a
function of model structure, problem objectives, data
constraints, and spatial and temporal scales of
application.
• Still searching for the unified theory of uncertainty
analysis.
Necessary Conditions for a Model to
be Considered Properly Calibrated
• Input-output behaviour of the model is
consistent with measured behaviour performance
• Model predictions are accurate (negligible
bias) and precise (prediction uncertainty
relatively small)
• Model structure and behaviour are
consistent with the understanding of reality
Gupta, H.V., et al, in review
National and
international groups
are collaborating to
assess existing
methods and tools for
uncertainty analysis
and to explore
potential avenues for
improvement in this
area.
http://www.es.lancs.ac.uk/hfdg/uncertainty_workshop/uncert_intro.htm
A Federal
Interagency
Working Group is
developing a
Calibration,
Optimization, and
Sensitivity and
Uncertainty
Analysis Toolbox
International Workshop Proceedings describes this effort:
available at http://www.iscmem.org
Future Model Development and Application
Coupled Hydrological Modelling Systems
Geochemical
Flowpaths
INPUTS
Model
Complexity
GIS
Landuse
Aquatic, Riparian
& Terrestrial
Hydrological
Modelling
Scale
PREDICTIONS
Uncertainty
Analysis
Increased Model Complexity
More Parameters
More Spatial Interactions
More Complex Responses
but still data limited ….
MORE MODELLING
UNCERTAINTY
Visual Uncertainty Analysis Framework
Modular Modelling
System - USGS
Deeper Valley
Zone
Bedrock
Depth (m)
5
4
3
2
1
0
Conceptualised
Bedrock
Model
Structures
Field
Measurements
Visualisations of
Models & Uncertainty
Visualisations
of Measurements
Improved
Representations of
Hydrological Processors
and Predictions
Freer, et al., Lancaster Univ., UK