Reflex paper 1
Reflex: Characterising uncertainty in analysis and prediction of
terrestrial C dynamics (using an intercomparison of model-data
fusion techniques)
Introduction
Situation – sources/sinks, future behaviour. Need for improvement illustrated by C4MIP (model
test), gap filling intercomparison.
Complication – feedback processes, net flux data, measurement uncertainty, model uncertainty and
failing of ordinary calibration approaches.
Solution – MDF, and in particularly the characterisation of uncertainty, normally through the
generation/use of parameter PDFs.
This paper aims to quantify the uncertainty associated with generating C model parameters (i.e.
isolating processes) from EC data and in making predictions of net C exchange.
This uncertainty stems from errors in a number of areas: optimisation algorithm, observations,
model, subjective (human error, related to set up), initial conditions and drivers.
Here we try to address 3 components:
Synthetic experiment - deals with observational and algorithmic error (and user error).
Real experiment – adds model error.
Hypotheses
Parameters linked to observations should be well characterised
Parameters for slow processes should be poorly characterised.
Algorithmic uncertainties on cumulative 2 year NEE should all be the SAME and represent our best
estimate of uncertainty.
Algorithmic uncertainty is the same in time (and similar to NEE observation error), and grows in
the prediction period.
Increased RMSE in forecast period – what is the variation?
Questions
How does intra- and inter-algorithm uncertainty compare?
What biases can be detected in parameterisation and forecast and can they be explained?
Performance of the algorithms – recommendations?
What is novel in this ms is…
- treatment of uncertainty
- process modelling, feedbacks between fluxes and stocks
- parameter estimation, dynamic retrieval, predictions with synthetic and real data
Methods
See protocol document.
Table A1. Detail of algorithms (id, description, user, key reference) – in Appendix a paragraph or
two of detail.
How were the confidence limits (CL) calculated by each group? What was the computational load?
1
Reflex paper 1
Results
1. Synthetic experiments
Figure 1. Parameter estimation. The top panel shows box and whisker plots of the algorithms’ best
estimate of each parameter, normalised by the true value used in the synthetic experiment. The
bottom panel shows box and whisker plots describing the magnitude of each algorithms’ 90%
confidence limit on each parameter estimate, normalised by the true value used in the synthetic
experiment. An alternative would be to plot actual values
Duplicate for DE (deciduous) and EV (evergreen)
Things to look for:
- are there any biases in the best estimates?
- which parameters have the largest uncertainties? – suggesting poorly constrained by data
- which mean estimates have the broadest range? – suggesting algorithm uncertainty.
Hypotheses:
We expect good mean estimates and small CLs for p2, p3,p5,p11,p12,p13,p15,p17
We expect poor mean estimates and large CLs for p1,p4,p6,p7,p8,p9,p10,p14,p16,
Table 1. Parameter estimation. For each parameter are shown its true values and also the mean+/90%CL estimated generated by each algorithm. Values in bold are consistent with the true value
(i.e. true value lies within the confidence limits).
Duplicate for DE (deciduous) and EV (evergreen).
Analyse this alongside figure 1 and decide which is best for the paper.
Probably too cumbersome for the paper, but useful for appendices.
Table 2 (not included at this stage). Parameter covariance (correlation?). Either an example
covariance/correlation matrix for parameters or something eslse? Are matrices of mean
values/standard deviations informative?
This requires further input at the moment.
Things to look for:
- Parameters that covary indicate where linked processes are poorly constrained.
- ??
Duplicate for DE (deciduous) and EV (evergreen)
Figure 2. Retrieval (years 1 and 2) and prediction (year 3) of C dynamics. Time series of NEE and
Cf showing truth, observations and mean and range of algorithm best estimates.
Duplicate for DE (deciduous) and EV (evergreen).
Need some basic time series to show obs, truth and analysis. There a number of possible
alternatives
Figure 3. Uncertainty on retrieval of cumulative/integrated C dynamics. (a) Time series of monthly
means (shows uncertainty between algorithms from range of means). (b) Monthly mean daily
uncertainty (90% CL size) for each month (shows uncertainty within each algorithm)
2
Reflex paper 1
Table 3. Retrieval of C dynamics. For each algorithm are shown the final cumulative (2 year and 3
yr) values and uncertainty for iNEE and large C pools. Also final row showing standard deviation
of the C values and mean of the uncertainties.
Things to look for:
- algorithm uncertainty (mean of the uncertainties on each algorithm) versus uncertainty among
algorithms (standard deviation of the best estimates of each algorithm).
- Uncertainty on slow turnover pools (all pools?)
Hypotheses:
Algorithmic uncertainties on cumulative 2 year NEE should all be the same and represent our best
estimate of uncertainty.
Algorithmic uncertainties is similar to NEE observation error.
There will be a growth in uncertainty in prediction year (relate to parameter and state observational
If error growth varies among methods then that reflects problems with error determination.
uncertainty)
Duplicate for DE (deciduous) and EV (evergreen).
Figure 4. Box and whisker plots of algorithm performance showing monthly RMSE for NEE
against (a) observations and (b) truth.
Table 4. Algorithm performance. For each algorithm, RMSE on NEE and Cf for analysis and
forecast for years 1 and 2 (analysis), and then year 3 (forecast) calculated using available
observations.
Table 5. Algorithm performance. For each algorithm, show mean daily RMSE on main fluxes and
stocks for analysis and forecast for years 1 and 2 (analysis), and then year 3 (forecast) calculated
using true values.
Things to look for:
-We expect increased RMSE in year 3 (forecast phase)
-Does this correspond with an increase in error estimated in table 3? Use figure 2 also to assist in
this.
Overall we are looking to find approaches that correctly estimate error – i.e. the measured NEE are
within the confidence intervals of the predicted NEE.
2. Real experiments
We are now dealing with model error as well (i.e. an inaccurate description of the system).
Duplicate figures and tables from synthetic experiment.
Figure 101. (Parameters and CLs are normalised by the mean of the parameter estimates).
We expect an increase in parameter uncertainty across the board, perhaps only 3-4 constrained
parameters.
Table 101. Compare with Figure 101. Which is most informative?
Table 102. Covariance matrix. We expect more divergence among algorithms and more correlation
and covariance due to model error.
3
Reflex paper 1
Figure 102 . We expect increased uncertainty in slow turnover parameters or stocks/fluxes remote
from NEE or LAI.
Figure 103. We expect larger uncertainty than the synthetic expt. due to model error, and larger
growth in error for all during prediction phase.
Table 103. We expect greater uncertainty on 2 yr and 3 yr cumulative NEE.
Figure 104.
Table 104. We expect increased RMSE in year 3 (forecast phase), does this correspond with an
increase in error estimated in table 3? Use figure 2 also to assist in this.
(Table 5 not possible as true values unknown)
4