This file was created by scanning the printed publication.
Errors identified by the software have been corrected;
however, some errors may remain.
A Method for Assessing the Prediction
Quality of Mechanistic Forest
Growth Models
Biing T. Guanl, George Gertner2and Pablo Parysow3
Abstract. A method for assessing the prediction quality of mechanistic
forest growth models was presented. The method consisted of four steps:
assuming distributions for parameter values, parameter screening,
outlining model behavior through sampling, and approximating model
behavior based on the sampled points. The proposed method was then
applied to a carbon balance stand level forest growth model. For the
example, a Monte Carlo method was used to perform the sampling, and
then a "patterned" artificial neural network was used for the
approximation. It was found that the predictions of the model were
relatively unbiased with respect to the initial parameter variances, and
the variances of the predictions were mainly contributed by only a few
parameters. Such information allows the model to be more closely
examined and provides a way to improve the model.
INTRODUCTION
For the past ten years, an important change in forest growth modeling has been
the shift in modeling approach from empirical modeling towards mechanistic (or
process) modeling; a shift that reflects the model developers ' desire to try to gain
better insights into how stands (or trees) grow and how future environment
changes may affect forest ecosystems (Bossel, 1991). Though still not having the
desired predictive power, mechanistic forest growth models are being developed
and used more frequently to answer some serious questions regarding the likely
impact of global environmental changes on various aspects of forest ecosystems.
An example is the use of TREGRO, a mechanistic tree growth model, to assess
the impact of 0, on seedling growth of red spruce (Laurence et al., 1993). Such a
usage of mechanistic models raises an important question: what is the prediction
quality (i.e., the amount of prediction bias and variance) of a mechanistic model
Associate Professor, Department of' Forestry, National Taiwan University, Taipei, Taiwan, Republic of China
Profexwr, Department of Natural Resources and Environmentul Sciences, University of Illinois at Urbana-Champaign
Urbana, IL
Ph. D. gmduate student, Department of Natural Resources and Environmental Sciences, University of Illinois at
Urbuna-Champaign, Urbana, IL
and how can such information be obtained? Existing prediction quality analysis
methods, such as a Monte Carlo approach or an error propagation approach ( e g ,
Gertner, 1987), cannot handle mechanistic models effectively.
In this study, a method is proposed for assessing the prediction quality of
mechanistic growth models. The framework of the method and layout of its key
steps will be described first. Then, as an example, the method is applied to a
mechanistic growth model. Finally, some comments will be made with respect to
the generality of the proposed method.
GENERAL FRAMEWORK O F THE METHOD
The core idea of the proposed prediction quality analysis method is as follows:
we believe that the sampling intensity required by a Monte Carlo approach can be
reduced through the use of a good approximation procedure while maintaining a
specified accuracy of the quality analysis. We should emphasize at this point that
the proposed method assumes that the target model is structurally and behaviorally
valid (see Bossel, 1992) for the intended purposes. The proposed method has the
following four key steps:
Assuming Ranges and Distributions: For the proposed method, we assume each
parameter in the model has a fixed range and a distribution within the specified
range. unlike empirical models, parameters in mechanistic models are usually not
estimated from data (some even argue that parameters in mechanistic models
should not be estimated, e.g., Bossel, 1992); they are usually derived from the
literature, small scale experiments, studies of related species, and from subjective
sources when information is sparse. Though often viewed as exact, parameter
values in mechanistic models should in reality be considered as interval estimates,
that is, the 'true" parameter value may be within a certain range, and the given
parameter value is the most likely a value within the range.
Screening Parameters: Parameter screening is necessary for reducing the actual
number of parameters that need to be examined. It is likely that in a mechanistic
model not all parameters contribute equally to the prediction quality (bias and
variance) of that model. Some parameters may have significant effects on
prediction quality, some may have slight effects, and still others may have no
effects. If those parameters that contribute little to prediction quality are screened
out, then computational resources can be saved in the subsequent steps.
Determining Monte Carlo Sampling Intensity: For the proposed method, two
sampling intensities need to be considered. The first intensity is the total number
of points (each point represents a combination of variabilities of the parameters
and the initial values of the state variables) that will be used in the subsequent
approximation. The sampled points roughly define the model ' s global behavior in
the sampled space. The second intensity is the number of runs that need to be
carried out at each sampled point to obtain actual prediction quality information at
that sampled point (i.e., the local behavior around the sampled point).
Selecting an Approximation Procedure: Together with the previous step, this
step determines the success of the proposed method. Choices abound for selecting
an approximation procedure. One can choose from simple polynomials to
complex nonlinear procedures, such as a spline or a neural network model.
Different approximation procedures offer different advantages.
EXAMPLE
The model that we used to demonstrate the proposed method is a mechanistic
forest growth model developed by Valentine (1988). The model is a carbon
balance model based on the pipe theory (Shinozaki et al., 1964 a,b) and the selfthinning rule. The model is a stand-level growth model for even-aged stands. It
has three state variables (basal area per unit area, active pipe length per unit area,
and total woody volume per unit area) and 20 parameters. In this example, the
pipe model was adapted for red pine (Pinus resinosa Ait.). The model was used to
simulate the growth of red pine for 30 years, from age 30 to age 60. For this
example, we will view the means as the \\trueu parameter values. Therefore, the
final projections based on those means will be treated as the "unbiased"
projections. Bias will thus be defined as the differences between the projected and
the "unbiased" values.
Step 1: For the example, the mean value and the lower and the upper bounds for
each of the parameters, as well as for the initial values of the state variables, were
directly taken from Parysow (1994). It is assumed that each parameter, as well as
the initial value of a state variable, has an unimodal and symmetric beta
distribution bounded between the prescribed lower and upper bounds. The
requirements on the shape of our beta distributions constrains the maximum
variance that our beta distributions can have. For a beta distribution to be
symmetric and unimodal, its two shape parameters need to be equal, and the
values must be greater than 1. In this example, it was decided that the minimum
value for the shape parameters of a beta distribution would be 1.5 which gives a
beta distribution with a well-defined mode. Under these constraints, the standard
deviation of a generic beta distribution in our analysis cannot be greater than 0.25;
in other words, the maximum standard deviation of a parameter distribution is 25
percent of its distribution range.
Step 2: Though the target model was a small model in terms of the number of
parameters and state variables, it was still desirable to screen out parameters that
had little influence on projection quality. The procedure used is Mann' s test for a
trend (Lehmann, 1975, pp. 290-297). Parameters were screened independently,
and the state variables were not included in the screening procedure. The a-level
to include was set to be 0.2 since the intention was to screen out the parameters
with little influences on the final projection quality.
Step 3: In this example, we chose a random sampling scheme to generate the
necessary sample points as well as the local behavior at each sampled point. Two
sets of data were generated. For the first set, the calibration data set, the sampling
intensity was 64K for the first level sampling and 16K for the second level
sampling. For the second set, the validation data set, the sampling intensities were
128K and 32K, respectively, for the first and second levels of sampling. It should
be noted that although parameter screening was performed (Step 2), a complete
sample was still done because the approximation results were used to validate the
screening results.
Step 4: The approximation procedure used in this example was a feedforward,
multilayer type of artificial neural network model. Artificial neural networks have
been proven both theoretically (CybenkoJ989; Hornik et al., 1989) and
empirically (e.g., Lapedes and Farber, 1987) to be excellent function
approximators under general conditions. Artificial neural networks have also been
used to approximate chaotic functions with good results (e.g., Lapedes and Farber,
1987). Though chaotic behavior might not exist in this example, it may exist in
other mechanistic models, especially the complex ones, since most of the
differential equations in existing mechanistic models have not been examined
thoroughly for their behaviors. For these reasons, artificial neural networks have
been chosen as the approximation tools in this example.
The networks used in this example all had an input layer, an output layer and a
hidden layer. The networks used were different from the typical feedforward
networks in two aspects. First, the nodes between the input layer and the hidden
layer were not fully connected, and were without the bias connections (Figure 1).
The nodes in the hidden layer were divided into groups, and each group was
connected only to one input node (groups were disjoint). Thus, each input was
trained by a subset of hidden nodes. The activation function for the hidden nodes
was the arc tangent (tan-l) function. The output activation function was just a
linear function (i.e., no squashing).
Figure 1. Network architecture for the proposed network.
Such a network architecture will produce outputs that can be partitioned
according to the contribution of each input. The connection pattern of the network
will isolate the contribution of each input, while the arc tangent activation
function will make some hidden nodes inactive (i.e., set to 0 ' s ) when the
corresponding inputs are inactive. The contributions from those inactive inputs to
the overall network outputs will thus be O f s. To be able to partition the outputs
according to the contribution of each input is important to the proposed procedure
since it allows us to examine the importance of each input and to develop an error
budget for the model.
The network we used in this example had five hidden nodes for each input
node. The training algorithm used in this case study was a modified version of a
random optimization procedure (Baba, 1989). The error function of the training
method was the squared error function (i.e., the squared difference between the
target and the actual outputs). Since this particular example involves a large
number of training and validation data, and the selected training algorithm is
parallel in nature, network training as well as validation were conducted on a
CM-2 parallel computer. Detail discussion of the training algorithm can be found
in Guan et al. (1993).
RESULTS AND COMMENTS
Since the results from the sampling showed that initial parameter variability did
not affect the projection accuracy (i.e., very little bias) of the pipe model, we will
focus on how initial parameter uncertainty affects the projection variances of the
pipe model.
Parameter Screening: The results from parameter screening confirm our initial
hypothesis that not all the parameters contribute significantly toward the
projection variance of the pipe model. Table 1 lists the parameters that the initial
screening procedure deemed to be significant (with p value, or calculated tail
probability, smaller than 0.2) to overall projection variance for each of the three
state variables. The entries are listed in a descending order according to the
calculated p value of each parameter.
Approximation Results: Table 2 gives the validation results of the trained neural
network for approximating the final projection variances of the three state
variables. The mean error rates are within 4 percent for all the three state
variables, and 99 percent of the approximations are within 14 percent of the target
values. It should be noted that the trained network was for approximating the
variances of the three state variables simultaneously (i.e., the network had three
output nodes).
Ranking the Importance of Parameters: Since the neural network model we
developed has the ability to isolate the variance contribution of an individual
parameter, we can use this feature to rank the importance of individual
parameters. Table 3 is an example of such a ranking. In the table, we can see that
given the same amount of relative initial variability, the one with the largest
contribution is the parameter S, a parameter that represents the effects of
environmental conditions on tree growth. What is interesting is that the first four
parameters in the table account for almost 75 percent of the total variability, and
the first ten parameters account for about 95 percent of the total projection
variance. The partitions of projection variances of the other two state variables
also show similar patterns, though with different parameter ranking. Common to
the projection variances of the three state variables is the relative importance of
the parameter S . Given the same amount of relative initial parameter variability,
this parameter accounts for roughly 37 percent, 63 percent and 55 percent of the
final projection variances for the state variables basal area, pipe length and total
Table 1. Results of parameter screening for the final projection variance for the three state
variables. Entries are listed in a descending order according to the calculated p value of each
parameter.
State Variable
Pipe Length
Basal Area
Total Volume
Table 2. Validation results for the trained neural network for predicting the final projection
variance of the three state variables. Numbers represents the relative error rates (in
percentage)
State Variables
Pipe Length
Basal Area
Total Volume
Mean
3.79
1.38
2.03
Minimum
25 Quantile
Medium
75 Quantile
95 Quantile
99 Quantile
Maximum
volume, respectively. Such a ranking in parameter contribution provides us with
clues to further improve the performance and projection quality of a model.
As the contribution of an individual parameter toward the overall prediction
quality becomes available, we can use such information to develop error budgets
for the model, to test hypotheses, and to analyze the statistical power of the tests,
steps that are all necessary in using mechanistic models to make informed
decisions.
Table 3. An example of a partition of the final projection variance of total basal area per unit
area based on the contribution of each parameter. The initial standard deviation for each
parameter was set at 5 % of the parameter Is range.
Parameter
Contribution
S
2.70
Relative
Importance (%)
37.52
Cumulative
Importance (%)
37.52
Basal Area
0.98
13.54
69.14
Pipe Length
0.29
3.98
85.57
Total Volume
0.09
1.22
95.57
Total
7.20
100.00
Comparing the rankings in Tables 1 and 3, not including the state variables, we
can see that the parameter screening procedure we proposed can indeed screen out
the parameters that have little influence on the final projection variance of total
basal area.The proposed screening procedure also was effective in screening out
the parameters that had little effect with respect to the other two state variables.
With a conservative screening criterion, we believe that the proposed screening
procedure can effectively reduce the number of parameters that need to be
considered while still maintaining the overall approximation accuracy. When the
model of interest is large and complex, parameter screening may save a large
amount of computation resources in the future.
As mentioned earlier, many approximation procedures can be used for the
proposed method, and one such method is orthogonal response surface analysis.
For the same pipe model, we have also developed a second-order orthogonal
response surface model based on a fractional factorial design (Gertner et al.,
1996). The orthogonal response model and the neural network model yielded
almost identical results, though the former requires less computation since it was
based on a fractional factorial design. However, when the model of interest has a
large number of parameters, or when the higher order approximation is called for
(e.g., the approximation is highly non-linear), then it is our belief that a neural
network approach may be more appropriate. A major difference between the two
models lies in the way the total projection variance (or bias) is partitioned. For an
orthogonal model, we may not be able to isolate the contribution of an individual
parameter since interaction with other parameters may be significant. Our neural
network model, on the other hand, has the ability to isolate the individual
parameter contribution, and the partition is exact (though not necessary
orthogonal).
The method proposed in this study is also general in the sense that the method
can be used as effectively in analyzing large and complex models as in small and
simple models, though complex or large models may require more computational
resources. For example, for complex multi-component models, we can adopt a
bottom-up, divide-and-conquer approach, that is, we analyze the behavior of
individual components, and then analyze the behavior of components toward the
overall model. Thus, it is our belief that the proposed method will contribute
significantly toward a more rational use of mechanistic models in ecological
research.
ACKNOWLEDGMENTS
The authors thanked the National Center for Supercomputing Applications,
University of Illinois at Urbana-Champaign, for granting the necessary computer
time on the CM-2 computer.
REFERENCES
Bossel, H., 1991. Modeling forest dynamics: Moving from description to
explanation. Forest Ecology and Management 42: 129-143.
Bossel, H., 1992. Real-structure process description as the basis of understanding
ecosystems and their development. Ecological Modelling 63: 261-276.
Cybenko, G., 1989. Approximation by superposition of a sigmoidal function.
Mathematics of Controls, Signals and Systems 2: 303-3 14.
Gertner, G., 1987. Approximating precision in simulation projections: an efficient
alternative to Monte Carlo methods. Forest Science 33: 230-239.
Gertner, G., P. Parysow and B. T. Guan. 1996. Projection variance partitioning of
a conceptual forest growth model with orthogonal polynomial. To appear in
Forest Science.
Guan, B. T. and G. 2. Gertner, and D. Kowalski. 1993. Modeling training site
vegetation coverage probability with a random optimization procedure: an
artificial neural network approach. In: the Proc. of the Conference of
Applications of Artificial Neural Networks IV, Orlando, Florida, pp. 682-688.
Hornik, K., Stinchcombe, M., and White, H., 1989. Multilayer feedforward
networks are universals approximators. Neural Network 2: 359-366.
Lapedes, A., and Farber, R., 1987. Nonlinear signal processing using neural
networks: Prediction and system modelling. Technical Report LA-UR-872662, Los Alamos National Laboratory, New Mexico.
Laurence, J. A., Kohut, R. J., and Amundson, R. G., 1993. Use of TREGRO to
simulate the effects of ozone on the growth of red spruce seedlings. Forest
Science 39(3): 453-464.
Lehmann, E. L., 1975. Nonparametrics: Statistical Methods Based on Ranks.
Holden-Day, San Francisco. 457 p.
Parysow, P. F, 1994. A procedure to approximate prediction uncertainty in
ecological conceptual models. Unpublished M.S. Thesis, University of Illinois
at Urbana-Champaign, Urbana, Illinois. 126 p.
Shinozaki, K., Yoda, K., Hozumi, K. and Kira, T., 1964a. A quantitative analysis
of plant form - the pipe model theory. I. Basic analysis. Japanese Journal of
Forest Ecology 14: 97- 105.
Shinozaki, K., Yoda, K., Hozumi, K., and Kira, T., 1964b. A quantitative analysis
of plant form - the pipe model theory. 11. Further evidence of the theory and its
application in forest ecology. Japanese Journal of Forest Ecology 14: 133-139.
Valentine, H., 1988. A carbon balance model of stand growth: a derivation
employing pipe-model theory and the self-thinning rule. Annals of Botany
621389-396.