LONG-TERM VARIATIONS IN THE PERFORMANCE OF CLIMATE MODELS
Alice M. Grimm 1
Atul K. Sahai 2
Chester F. Ropelewski 3
ABSTRACT: The skill of numerical climate models, usually considered to be constant for a given
season, is shown in this study to undergo interdecadal variations. The relationships between these
skill variations and interdecadal modes of climate variability are examined through Empirical
Orthogonal Function analyses of the sliding correlations between observed and modeled fields, and
through Influence Function analysis. Interdecadal variations in model skill reflect long-term
variability in ocean-atmosphere links, and in regional teleconnections. This suggests un-modeled
climate processes that affect seasonal climate prediction as well as greenhouse-gas scenarios. The
reliability of these scenarios may depend on the time slice being analyzed.
RESUMO: Neste estudo, demonstra-se que o desempenho de modelos numéricos climáticos,
usualmente considerado constante para uma dada estação do ano, sofre variações interdecadais. As
relações entre as variações de desempenho e modos interdecadais de variabilidade climática são
examinadas através de análise de Funções Empíricas Ortogonais das correlações móveis entre
campos observados e simulados, além de análise com Funções de Influência. Variações
interdecadais no desempenho do modelo refletem variabilidade de longo período nas conexões
oceano-atmosfera e nas teleconexões regionais. Isto sugere a existência de processos climáticos não
adequadamente modelados que afetam tanto a previsão climática sazonal como cenários de
mudanças climáticas. A confiabilidade destes cenários pode depender do período de anos a ser
analisado.
1
Universidade Federal do Paraná - Depto de Física, Caixa Postal 19044, 81531-990 Curitiba, PR, Fone: 41 361-3097,
grimm@fisica.ufpr.br
2
Indian Institute of Tropical Meteorology, Pune, India, sahai@tropmet.res.in
3
International Institute for Climate Prediction, Palisades, New York, chet@iri.columbia.edu
Assessing the ability of dynamical atmospheric general circulation models (AGCMs) to
reproduce observed atmospheric circulation given the lower boundary conditions, and thus its
ability to predict climate, has been a recurrent concern in seasonal-to-interannual climate prediction.
Assessments have been carried out in several ways including comparisons between the leading
modes of observed variability and those simulated by models (1). In several of those studies it is
shown that the performance of models is seasonally-dependent. This is related to seasonal variations
in the atmosphere modes of variability as well as seasonal variations in the ability of the ocean
boundary to influence a particular region. However, there has always been the assumption that
model skill is constant for a given season throughout the period being analyzed.
Several studies have shown that the atmosphere and the oceans undergo interdecadal
variations (2, 3). However, the influence of multi-year e.g., interdecadal variability, on model
performance has not been addressed. In this study we examine long-term variability in the seasonal
response of two numerical models driven by observed sea surface temperature (SST).
Two AGCMs are examined in conjunction with observations for the period 1950 to 1994. The
ECHAM3 (version 3.6, Max Planck Institute) and the NCEP (MRF9, National Centers for
Environmental Prediction) model have comparable spatial resolution (4), but several differences in
their parameterization schemes (5, 6).
Both models are forced by reconstructed observed SST for the lower boundary (7, 8, 9). The
ensemble means of seven runs of the ECHAM3 model and thirteen runs of the NCEP model are
examined. Observations are based on the NCEP-NCAR reanalysis (10). Although the NCEP-NCAR
reanalysis and the NCEP model are versions of the same model, there are considerable differences
in their resolution and parameterization schemes (1).
Interdecadal variability of model performance is assessed through analysis of simultaneous
correlation coefficients (CCs) between the reanalysis data and the model output, averaged over
every 20º latitude × 40º longitude region over the globe (11). Seasonal correlations are computed
for boreal winter (December to February) in overlapping 11-year running periods (12). The
temporal variation of these CCs is a measure of interdecadal variation of the models’ skill all over
the globe.
We concentrate on the analysis of sliding correlation coefficients between observed and
modeled 200 hPa streamfunction, which is, on the average, the best simulated parameter globally,
according to our assessment. When comparing both models, Fig. 1a and 1b, it is evident that there
are more similarities than differences in the time series of the sliding correlations between observed
and modeled streamfunction. In spite of the differences in overall skill, the two models point
coherently to interdecadal variations of their performance. This coherence is somewhat better in the
Southern Hemisphere than in the Northern Hemisphere (13), perhaps reflecting the fact that the
NCEP-NCAR Reanalysis data set in most of the Southern Hemisphere is basically result of
modeling, due to the relatively sparse observational data.
We also examined the long-term variations of the models´ performance by comparing the
temporal variability of the observed response of the atmosphere to ENSO with the temporal
variability of the models´ response to ENSO. Correlations of 11-year running series of SST in the
Niño 3 region (5S-5N; 90W-150W) and observed (NCEP Reanalysis) seasonal 200 hPa
streamfunction are compared with the corresponding sliding correlations of Niño 3 SST and
streamfunction output from each AGCM. The variations in the CCs between the observed
streamfunction and the Nino 3 SSTs provide as estimate of the temporal variations in the strength of
ocean-atmosphere coupling. The temporal variability in the observed response of the atmosphere to
ENSO, as well as in the models´ response, is shown in Fig. 2. The correlation for the DecemberJanuary-February (DJF) season undergoes stronger variations throughout the years in the
observations, indicating interdecadal variability in the ocean-atmosphere climate link (Fig. 2a), than
in the models (Fig. 2b, c). These variations seem to be a little stronger in DJF than in June-JulyAugust (JJA), not shown. The ECHAM3 model reproduces the observed changes in that
relationship more faithfully than the NCEP model, especially in the American and Atlantic sectors
(Figs. 2b). In general, the temporal variability of the sliding correlations in the NCEP model is less
than in the ECHAM3 (Fig. 2c).
The relationships between interdecadal modes of SST variability and the interdecadal
variations of the models’ skill are examined through Empirical Orthogonal Function (EOF) analyses
of the sliding correlations between observed and modeled fields, displayed in Fig. 1. Correlations of
the first two principal components with SST give an indication of the relationship between the
interdecadal variability of the models’ skill and the interdecadal variability of SST. The statistical
significance of the correlations is assessed by using a Monte Carlo procedure (14).
The analysis is shown for the Northern Hemisphere (NH) winter (DJF) because in this season the
SST main modes of variability are considered to have larger components and the tropospheric
associations are stronger (e.g., 2, 15). EOF analysis for the NH spring (March-April-May) show
similar patterns but the factor loadings of the first two modes are weaker. We only show results
from the ECHAM3 model.
The first principal component explains 46 % of the variance, and its factor loadings (Fig. 3a)
feature a fairly zonal pattern with opposite signs in the equatorial belt and in the extratropics. In the
equatorial zone loadings project heavily on the central-eastern Pacific and on the eastern Indian
Ocean. There are zonal asymmetries, especially in the NH, as if following a Rossby wavetrain. The
second principal component explains 26 % of the variance and also shows a dominant zonal
distribution of the factor loadings (Fig. 3b), alternating opposite signs in the equatorial belt, the
subtropics and the higher latitudes. As in the first mode, there are also zonal asymmetries like those
produced by Rossby wavetrain propagation. According to the principal components of these two
modes (Fig. 3c) the first mode undergoes interdecadal variation, with a transition of phase in the
70´s, while the second mode shows a nearly decadal timescale. The modes obtained from the
correlation with the NCEP model (not shown) present the same general characteristics, though there
are some differences in the magnitude of the factor loadings and the location of the maximum
values.
To understand possible physical mechanisms behind the variability of the models´
performance, the first and second PCs were correlated with SST averaged over 2040 latitudelongitude regions. The patterns of correlation indicate regions where SST varies coherently with the
modes of models´ performance. The correlation coefficients with the first PC feature higher
correlation in the North Pacific and the North Atlantic, with correlation of opposite sign prevailing
in the Southern Hemisphere (not shown). These patterns resemble the spatial patterns of SST
variability associated with the Atlantic Multi-decadal variability and some features of Pacific Multidecadal variability, which are two reported modes of non-ENSO low-frequency SST variability (2,
16).
The correlation coefficients between global SST and the second PC display patterns
resembling those of another non-ENSO mode of low-frequency variability of SST reported in (2),
the Pacific Interdecadal mode. Besides reproducing the strongest components of this mode in the
subtropics of the eastern Pacific Ocean, particularly off the southwest US coast, the spatial
distribution of correlation also reproduces the spatial distribution of that mode in the North Atlantic
and in the North and South Pacific. The correspondent temporal series of that mode also shows
significant resemblance to PC2 of Fig. 3c during the period analyzed.
The correlation coefficients between SST and PC1 and PC2 obtained using the NCEP model
output (not shown) have a similar distribution. This similarity suggests that, in spite of the
differences between the performances of the two models, the results concerning the interdecadal
variations of the performance are robust.
The interdecadal fluctuations of the ability of the AGCMs forced with observed SST in
reproducing the variability of the atmosphere seem to reflect fluctuations in the ocean-atmosphere
links caused by the interdecadal climate variability as well as the inability of the models to operate
properly in climate regimes different from those for which they were tuned. This suggests
unmodeled climate processes, which have serious implications for seasonal climate prediction as
well as greenhouse-gas scenarios.
Given the use of these models in seasonal climate predictions and similar models in scenario
generation for global change studies, several aspects should be analyzed further, regarding the
variations in the model skill: 1) the model may not be able to reproduce correctly the anomalous
tropospheric heat sources (resulting from convection) associated with interdecadal modes of
variability (17). The misrepresentation of these sources might impact significantly on the
atmospheric circulation, both in the tropics and in the extratropics. 2) The models may not be able
to reproduce the interdecadal changes in the basic state of the atmosphere, which may modify their
ability to represent well some processes, like the propagation of Rossby waves generated by tropical
heat sources into the extratropics. The basic state in which the perturbations develop and propagate
is important for their evolution. 3) The decrease/increase in the performance of the models during
extreme phases of interdecadal modes of variability may be due to changes of the basic state of the
atmosphere. The best adjustment of a model for reproducing observed fields and their variations
during one phase of an interdecadal oscillation might not be best one during the opposite phase.
The influence of the atmospheric basic state in the propagation of Rossby waves is
demonstrated by an analysis of the Influence Functions of a vorticity equation model at 200 hPa,
which includes the divergence of the basic state and the vorticity advection by the anomalous
divergent wind (18). Figure 4 shows the December influence function for a target point in southern
South America computed with basic states for two different periods: 1950-1972 and 1972-1994.
The amplitude of the Influence Function is much stronger in the second period than in the first in
the equatorial region, and even the signs are opposite, indicating that upper-level divergence
(associated with heat sources) in this region was much more efficient in producing rotational
circulation anomalies around the target point in the more recent decades.
The implications of these results are that if a model is well adjusted for a given period, it will
not necessarily perform equally well in another period. Therefore, caution is recommended in the
use of long model runs for studying the regional effect of increasing greenhouse gases. Changes in
the basic state of the atmosphere-ocean system may occur naturally as well as part of greenhousegas related variations. Influence Function analysis demonstrates that changes in the basic state
result in variations in the atmospheric teleconnections. The analysis presented here suggests that
models do not adequately reproduce the interdecadal changes in the basic state. Thus the regional
reliability of long climate model runs may depend on the time slice in which the output of the model
is analyzed.
Acknowledgments: This study was supported by CNPq-Brazil and IAI (CRN-055)
REFERENCES AND NOTES
1. Peng, P., A. Kumar, A. G. Barnston, L. Goddard, 2000: Simulation skills of the SST-forced
global climate variability of the NCEP-MRF9 and the Scripps_MPI ECHAM3 models. J. Climate,
13, 3657-3679.
2. Enfield, D. B., and A. M. Mestas-Nuñez, 1999: Multiscale variabilities in global sea surface
temperatures and their relationships with tropospheric climate patterns. J. Climate, 12, 2719-2733.
3. Vimont, D. J., D. S. Battisti, and A. C. Hirst, 2002: Pacific Interannual and Interdecadal
equatorial variability in a 1000-year simulation of the CSIRO Coupled General Circulation Model,
J. Climate 15, 160-178.
4. The NCEP model has truncation T40 with 18 sigma levels; ECHAM3 has T42 truncation with 19
sigma levels.
5. Deutsches Klimarechenzentrum, 1992: The ECHAM-3 Atmospheric General Circulation Model,
Tech. Rep. 6, ISSN 0940-9327, 189 pp. [Available from the Modellbetreuungsgruppe, Deutsches
Klimarechenzentrum, Max Planck Institute für Meteorologie, Bundestr. 55, D-20146, Hamburg,
Germany.]
6. Kumar, A., M. Hoerling, M. Ji, A. Leetmaa, and P. Sardeshmukh, 1996: Assessing a GCM’s
suitability for making seasonal predictions. J. Climate, 9, 115-129.
7. Reynolds, R. W., and T. M. Smith, 1994: Improved global sea surface temperature analysis using
optimum interpolation. J. Climate, 7, 929-948.
8. Smith, T. M., R. W. Reynolds, R. E. Livezey, and D. C. Stokes, 1996: Reconstruction of
historical sea surface temperatures using empirical orthogonal functions. J. Climate, 9, 1403-1420.
9. Although the NCEP model used the optimum interpolation (OI) data (7) for the 1982-94 period.
The two datasets are very similar in this period.
10. Kalnay E., M. Kanamitsu, R. Kistler, W. Collins, D. Deaven, L. Gandin, M Iredell, S. Saha.
G.White, J. Wollen, Y Zhu, M. Chelliah, W. Ebisuzaki, W. Higgins, J. Janowiak, K.C. Mo, C.
Ropelewski, J. Wang. A. Leetmaa, R. Reynolds, Roy Jenne and Dennis Joseph, 1996: The
NCEP/NCAR 40-year reanalysis project. Bull. Amer. Met Soc. 77, 437-471.
11. It is recognized that the procedure of averaging over 20º × 40º regions gives a poor spatial
resolution, but it allows a comprehensive view of variations in the models’ performance.
12. The contrasting seasons of DJF and JJA were analyzed, to stress the interseasonal differences,
but only the DJF results are shown here.
13. The comparison between results for different seasons, but considering the same model and
parameter, discloses more differences than similarities. This means that the fluctuations of the
models’ performance are seasonally dependent.
14. 10,000 random permutations of the SST field data are generated and the CCs with the principal
component series are calculated. Then the number of times in which the absolute value of the CC
with the random permutations exceeds that with the original data is counted. This value, divided by
10,000, gives the level of significance.
15. Venegas, S. A., L. A. Mysak, and D. N. Straub, 1997: Atmosphere-ocean coupled variability in
the South Atlantic. J. Climate, 10, 2904-2920.
16. As the period analyzed in this study is shorter than that used by Enfield and Mestas-Nuñez, it is
not possible to separate these two modes, as they have some common features, mainly in the Pacific
Ocean, and both have a change of phase in the 70’s.
17. Allan, R. P., and Slingo, A, 2002: Can current climate model forcings explain the spatial and
temporal signatures of decadal OLR variations? Geophys. Res. Lett., 29, art. no. 1141.
18. Grimm, A. M., and P. L. Silva Dias, 1995: Analysis of tropical-extratropical interactions with
influence functions of a barotropic model. Journal of the Atmospheric Sciences, 52, 3538-3555.
Figure 1 (Top) Simultaneous 11-year running correlation coefficients (CC) between the DJF 200 hPa streamfunction
from the NCEP/NCAR reanalysis and from the ECHAM3 model output, in the period 1950-1994. Before calculating
the CC, values were averaged over 2040 latitude-longitude regions. In each grid region, values of CC (from –1 to
+1) are on the Y-axis and the central years of the sliding windows (from 1955 to 1989) are on the X-axis. The threshold
for two-sided statistical significance at the 0.05 level is 0.60. (Bottom) Same as above for correlations between the DJF
200 hPa streamfunction from the NCEP/NCAR reanalysis and from the NCEP T40 model output.
Figure 2 (Top) Simultaneous 11-year running correlation coefficients (CC) between the Niño 3 SST index and the 200
hPa streamfunction from the NCEP/NCAR reanalysis, in DJF of the period 1950-1994. Before calculating the CC,
values were averaged over 2040 latitude-longitude regions. In each grid region, values of CC (from –1 to +1) are on
the Y-axis and the central years of the sliding windows (from 1955 to 1989) are on the X-axis. The threshold for twosided statistical significance at the 0.05 level is 0.60. (Middle) Same as above for CC between the Niño 3 SST index
and the 200 hPa streamfunction from the ECHAM3 model output. (Bottom) Same as above for CC between the Niño 3
SST index and the 200 hPa streamfunction from the NCEP T40 model output.
a.
b.
c.
Figure 3. (Top) Factor loadings of the first EOF of the 11-year running correlation coefficients between the DJF 200
hPa streamfunction from the NCEP/NCAR reanalysis and from the ECHAM3 model output, in the period 1950-1994
(35 running decades).(Middle) same as above for the second EOF. (Bottom) First two PCs of the 11-year running
correlation coefficients between the DJF 200 hPa streamfunction from the NCEP/NCAR reanalysis and from the
ECHAM3 model output, in the period 1950-1994 (35 running decades), shown in Fig. 1(Top).
a
b
Figure 4. (Top) Influence function for a target point in southern South America (indicated by a black circle) for the
December basic state of the period 1949-1972. (Bottom) Same as above, for the December basic state of the period
1972-1994. The values indicate the efficiency of the upper level divergence (which in the tropics is usually associated
with tropospheric heat sources) in producing streamfunction anomalies around the target point.