Testing modeling frameworks
or…
how can we improve predictive skill
from our data?
Mary C. Hill, University of Kansas; Emeritus, U.S. Geological Survey
Laura Foglia, Technical University of Darmstadt (soon at UC Davis)
Ming Ye, Florida state University
Lu Dan, Oak Ridge National Laboratory
Frameworks for Model Analysis
• Look for surprises
• Surprises can indicate
New understanding about reality
or
Model limitations or problems
• Fast methods are convenient -- enable routine
evaluation. Computationally frugal, parallelizable
methods
What do I mean by a modeling
framework?
• Methods used to
– Parameterize system characteristics
– Compare to data
– Adjust for data (calibration, tuning)
– Sensitivity analysis
– Uncertainty quantification
What do I mean by a modeling
framework?
• Methods used to
– Parameterize system characteristics
Define zones
or simple
interpolations
and estimate
relatively few
parameter
values
Quaternary
geology
map
X
Conductivity
map
9
Define many
interpolation
points.
Estimate a
parameter
value at each
point.
What do I mean by a modeling
framework?
• Methods used to
– Parameterize system characteristics
– Compare to data
• Graphical
• Statistics like sum of squared residuals
What do I mean by a modeling
framework?
• Methods used to
– Parameterize system characteristics
– Compare to data
– Adjust for data (calibration, tuning)
What do I mean by a modeling
framework?
• Methods used to
Foglia et al, in preparation
0.5
425
0.4
390
0.3
355
0.2
320
0.1
285
0.0
250
Head identifier
Head, in meters
Observation importance
using leverage, Cook’s D
– Parameterize system characteristics
– Compare to data
– Adjust for data (calibration, tuning)
– Sensitivity analysis
What do I mean by a modeling
framework?
• Methods used to
– Parameterize system characteristics
– Compare to data
– Adjust for data (calibration, tuning)
– Sensitivity analysis
– Uncertainty quantification
Model runs
100
1,600
480,000
True
Lu et al. 2012 WRR
HO
3Z
INT
Name of alternative model……..
Overview of
framework
methods
Problems:
-Many method - Tower of
Babel?
-Colossal computational
demands – Necessary?
Hill + 2015, Groundwater
Testing modeling frameworks
Test using cross-validation
Maggia Valley,
Switzerland


Goal: Integrated
hydrologic model to help
manage ecology of this
altered hydrologic system.
Use component sw and gw
models to test methods
for the integrated model.
1954: Dams closed upstream
11
Maggia Valley, southern Switzerland
Series of studies to identify and test a computationally frugal
protocol for model development
1. Test frugal sensitivity analysis (SA) with cross-validation
– MODFLOW Model w stream (dry) (Foglia+ 2007 GW)
2. Demonstrate frugal optimal calibration method
– TOPKAPI Rainfall-Runoff model (Foglia + 2009 WRR)
3. Here - SA and calibration methods enable test of
computationally frugal model selection criteria
MODFLOW GW model (wet) (Foglia + 2013 WRR)
4. Integrated hydrologic model – SW and GW
Software: UCODE (Poeter + 2005, 2014)
MMA (MultiModel Analysis) (Poeter and Hill 2007)
12
Model outline and 35 Obs Locations
Flow data (use gains/losses)
Head data
13
13
Model structures
SFR
RIV
• Streamflow Routing (SFR2) vs.
River (RIV)
• Recharge
– Zero, constant, TOPKAPI
• Bedrock boundary
– Old & New: Before and after
gravimetric survey.
– Depth in south x 2.
Quaternary
geology
map
K
• K
– Based on map of surficial
quaternary geology
– 5 combinations, from 1 parameter
to 6
64 alternative models
Range:
5e-4 to
1e-3
Conductivity
map 14
Cross validation experiment
Flow data (use gains/losses)
3 gains/losses
omitted for
cross-validation
experiment
Head data
8 heads omitted for
cross-validation
experiment
15
15
Use SA measure leverage for models calibrated with all 35 obs.
High leverage = important obs.
1.0
0.9
0.8
R4new_2
Each
symbol
R2new_3
represents
a
R4new_3
R2new_2model
different
R2old_2
0.7
R4new_4
S1old_5 (KIC 1)
0.6
S4old_5 (KIC 2)
S2old_5 (KIC 3)
0.5
S1old_4 (KIC 4)
S4new_5 (KIC 5)
0.4
0.3
0.2
0.1
0.0
11
22
33
44
55
66
77
88
99
10
10
11
11
12
12
13
13
14
14
15
15
16
16
17
17
18
18
19
19
20
20
21
21
22
22
23
23
24
24
25
25
26
26
27
27
28
28
V1_diff
V1_diff
R1_diff
S1_diff
R1_diff
S2_diff
S1_diff
S3_diff
S2_diff
L1_diff
S3_diff
A1_diff
L1_diff
Leverage
Leverage
How important are the omitted observations in
the alternative models?
Head observation number and flow gain/loss observation name
Observation number or name
16
How important are the omitted observations in
the alternative models?
1.0
methods of analyzing alternative models
R4new_2
0.9
R2new_3
R4new_3
0.8
R2new_2
R2old_2
0.7
R4new_4
S1old_5 (KIC 1)
0.6
S4old_5 (KIC 2)
S2old_5 (KIC 3)
0.5
S1old_4 (KIC 4)
S4new_5 (KIC 5)
0.4
0.3
0.2
0.1
0.0
11
22
33
44
55
66
77
88
99
10
10
11
11
12
12
13
13
14
14
15
15
16
16
17
17
18
18
19
19
20
20
21
21
22
22
23
23
24
24
25
25
26
26
27
27
28
28
V1_diff
V1_diff
R1_diff
S1_diff
R1_diff
S2_diff
S1_diff
S3_diff
S2_diff
L1_diff
S3_diff
A1_diff
L1_diff
Leverage
Leverage
Omitted
obs
areSAvery
important
toIdentifies
not important.
Quickly
evaluate
using
measure
leverage.
observations
important
and unimportant
to parameters.
Omitting
these obs
should provide
a good test of
Head observation number and flow gain/loss observation name
Observation number or name
17
Tests conducted
• Test model selection criteria AIC, BIC, and KIC
– Forund problems – especially for KIC
• Compared results with a basic premise related
to model complexity
– Led to some specific suggestions about model
development
– Present this part of the analysis here
12
Observation
misfit (o) 10
or
8
Prediction
6
error ( )
We want this: the
best predictions
Prediction error
1200
1000
800
600
400
200
0
130
4
Model Fit
2
0
1
10
100
Number of parameters
1000
Model fit (SSWR)
Discrepancy between model
and observations ( o) or
Prediction error ( x)
Our results vs basic
premise?
Prediction error (SSWpred)
1400
110
90
70
50
30
4
Lines connect models that
are the same except for K
5
6
7
8
9
10
Number of estimated parameters
19
12
Observation
misfit (o) 10
or
8
Prediction
6
error ( )
We want this: the
best predictions
Prediction error
1200
1000
800
600
400
200
0
130
4
Model Fit
2
0
1
10
100
1000
Number of parameters
Lack of improvement with added K
parameters indicates the K
parameterization has little relation to the
processes affecting the data
Model fit (SSWR)
Discrepancy between model
and observations ( o) or
Prediction error ( x)
Our results vs basic
premise?
Prediction error (SSWpred)
1400
110
Solid: SFR
Hollow: RIV
90
70
50
30
4
Lines connect models that
are the same except for K
5
6
7
8
9
10
Number of estimated parameters
20
12
Observation
misfit (o) 10
or
8
Prediction
6
error ( )
We want this: the
best predictions
Prediction error
1200
1000
800
600
400
200
0
130
4
Model Fit
2
0
1
10
100
Number of parameters
1000
Model fit (SSWR)
Discrepancy between model
and observations ( o) or
Prediction error ( x)
Our results vs basic
premise?
Prediction error (SSWpred)
1400
110
Solid: SFR
Hollow: RIV
90
70
50
30
4
Lines connect models that
are the same except for K
5
6
7
8
9
10
Number of estimated parameters
21
12
Observation
misfit (o) 10
or
8
Prediction
6
error ( )
We want this: the
best predictions
Prediction error
1200
1000
800
600
400
200
0
130
4
Model Fit
2
0
1
10
100
Number of parameters
1000
Model fit (SSWR)
Discrepancy between model
and observations ( o) or
Prediction error ( x)
Our results vs basic
premise?
Prediction error (SSWpred)
1400
110
Solid: SFR
Hollow: RIV
90
70
50
30
4
Lines connect models that
are the same except for K
5
6
7
8
9
10
Number of estimated parameters
22
12
Observation
misfit (o) 10
or
8
Prediction
6
error ( )
We want this: the
best predictions
Prediction error
1200
1000
800
600
400
200
0
130
4
Model Fit
2
0
1
10
100
1000
Number of parameters
Sensitivity analysis indicated the models
with the most parameters were poorly
posed. Suggests we can use these
methods to detect poor models.
Model fit (SSWR)
Discrepancy between model
and observations ( o) or
Prediction error ( x)
Our results vs basic
premise?
Prediction error (SSWpred)
1400
110
Solid: SFR
Hollow: RIV
90
70
50
30
4
Lines connect models that
are the same except for K
5
6
7
8
9
10
Number of estimated parameters
23
What do poor predictions look like?
(a)
3
2
1
0
-1
-2
-3
-7.3
Number of head observation
12
11
10
9
8
7
6
-4
5
Observed minus predicted value, in m
4
S4old_2
S4old_3
R1old_2
R2new_3
R4new_3
S2old_3
R4new_4
S4old_5
R4old_2
S4new_2
R4new_2
R4old_3
S2new_2
R2old_2
R2new_2
S2new_3
S2old_5
R2old_5
S4new_5
S1old_4
Identify important parameters
Results from the integrated hydrologic model
Groundwater model + distributed hydrologic model +
new observations for low flows
Original R-R model
Observations
120 streamflows
New integrated model
618 streamflows
28 GW head observations
7 gain/loss observations
Parameters included in
calibration
Processes
Results
4 SW parameter types
6 SW parameter types (36 pars)
3 GW parameter types (6 pars)
Rainfall-runoff model
Rainfall-runoff model
Groundwater model
More frequent sampling for Low flows are still as important as
low flows is needed (Foglia + high flows for many of the
2009 WRR)
parameters
Observations related to the
groundwater model are important
also for the rainfall-runoff model
B_ETP1
depth2
depth3
depth4
depth5
depth8
depth11
KS2
KS3
KS4
KS5
KS8
KS11
thetas2
thetas3
thetas4
thetas5
thetas8
thetas11
MAN_ovr1
MAN_ovr2
MAN_ovr3
MAN_ovr4
MAN_ovr5
MAN_ovr6
MAN_ovr7
MAN_ovr8
MAN_ovr9
MAN_ovr10
MAN_ovr11
MAN_ovr12
MAN_RIV1
MAN_RIV2
MAN_RIV3
MAN_RIV4
MAN_RIV5
KRIVER1
KRIVER2
K2
K57
K6
RCH_1
Composite Scaled Sensitivity
Identify important parameters
14
12
10
8
GW Heads obs
GW flow obs
SW low-medium flows
SW high flows
B_ETP1
Soil
depth
Saturated
K
Porosity
TOPKAPI
PARAMETERS
Overland
Mannings
Rivers
MODFLOW
PARAMETERS
6
4
2
0
KRiv
K Rech
Identify important parameters
GW observations are quite important to some of the SW parameters
SW obs are interpreted as GW flow (gain/loss) obs, which are important
Composite Scaled Sensitivity
14
GW Heads obs
GW flow obs
SW low-medium flows
SW high flows
12
10
8
TOPKAPI
PARAMETERS
MODFLOW
PARAMETERS
6
4
2
B_ETP1
depth2
depth3
depth4
depth5
depth8
depth11
KS2
KS3
KS4
KS5
KS8
KS11
thetas2
thetas3
thetas4
thetas5
thetas8
thetas11
MAN_ovr1
MAN_ovr2
MAN_ovr3
MAN_ovr4
MAN_ovr5
MAN_ovr6
MAN_ovr7
MAN_ovr8
MAN_ovr9
MAN_ovr10
MAN_ovr11
MAN_ovr12
MAN_RIV1
MAN_RIV2
MAN_RIV3
MAN_RIV4
MAN_RIV5
KRIVER1
KRIVER2
K2
K57
K6
RCH_1
0
B_ETP1
Soil
depth
Saturated
K
Porosity
Overland
Mannings
Rivers
KRiv
K Rech
Conclusions from this test
• The surficial geologic mapping did not provide
a useful way to parameterize the K
• Sensitivity analysis can be used to determine
when the model detail exceeds what can be
supported by data.
• Models unsupported by data are likely to have
poor predictive ability
• 5 suggestions for model development
Testing modeling frameworks
• Enabled by computationally frugal methods,
which are enabled by computationally robust
models
• This is where you, the model developers come
in!
Do your models have
numerical daemons???
Kavetski Kuczera 2007 WRR; Hill + 2015 GW; test with DELSA Rakovec 2014 WRR
Need important nonlinearities. Keep them!
Get rid of unrealistic, difficult numerics
Identify important parameters
GW observations are quite important to some of the SW parameters
SW obs are interpreted as GW flow (gain/loss) obs, which are important
Composite Scaled Sensitivity
14
GW Heads obs
GW flow obs
SW low-medium flows
SW high flows
12
10
8
TOPKAPI
PARAMETERS
MODFLOW
PARAMETERS
6
4
2
B_ETP1
depth2
depth3
depth4
depth5
depth8
depth11
KS2
KS3
KS4
KS5
KS8
KS11
thetas2
thetas3
thetas4
thetas5
thetas8
thetas11
MAN_ovr1
MAN_ovr2
MAN_ovr3
MAN_ovr4
MAN_ovr5
MAN_ovr6
MAN_ovr7
MAN_ovr8
MAN_ovr9
MAN_ovr10
MAN_ovr11
MAN_ovr12
MAN_RIV1
MAN_RIV2
MAN_RIV3
MAN_RIV4
MAN_RIV5
KRIVER1
KRIVER2
K2
K57
K6
RCH_1
0
B_ETP1
Soil
depth
Saturated
K
Porosity
Overland
Mannings
Rivers
KRiv
K Rech
Perspective
• In 100 years, we may very well be thought of
as being in the Model T stage of model
development
• A prediction: In the future, simply getting a
close model fit will not be enough. It will also
be necessary to show what obs, pars, and
processes are important to the good model fit.
• This increased model transparency is like the
safety equipment in automobile development.
I am looking for a test case
• Test a way to detect and diagnose numerical
daemons.
• High profile model applications preferred.
AGU: New Technical Committee on
Hydrologic Uncertainty
SIAM: New SIAM-UQ section and journal
Quantifying Uncertainty
• Lu Ye Hill 2012 WRR
• Investigate methods that take 10s to 100,000s
of model runs (Linear to MCMC)
• New UCODE_2014 with MCMC supports these
ways of calculating uncertainty intervals
Test Case
•
•
•
•
•
MODFLOW
3.8 km x 6 km x 80m
Top: free-surface
No-flow sides, bottom
Obs: 54 heads, 2 flows,
lake level, lake budget
• Parameters:
(b)
(a)
(c)
5 for
• recharge
• KRB
• confining unit KV
• net lake recharge,
• vertical anisotropy
K : 1 (HO), 3 (3Z), or 21 (INT)
Data on K at yellow dots
True
(d)
3Z
Int
Calibrated model, Predictions, Confidence Intervals
Model runs
100
1,600
480,000
True
HO
3Z
INT
Name
of alternative
Statistics to use when
the truth
is unknown model……..
HO
Lesson about uncertainty
3Z
Int
error
of
1.49
1.00
 May Standard
be more
important
to consider1.27
the uncertainty
regression
(1.25-1.84)
(1.06-1.57)
(0.88-1.30)
from
many models than0.54
use a computationally
1Intrinsic nonlinearity
0.02
0.26
demanding
R 2way to calculate uncertainty for one model
N