Appendix
II
Technical
Report:
Evaluation
of
APCC
Retrospective Forecasts for seasonal precipitation
In this technical report, we focus on evaluating one-month lead seasonal
prediction of precipitation in APCC retrospective multi-model ensemble (MME)
forecast system. The participating models in APCC include six two-tier and one
one-tier prediction systems from China, Japan, Korea, Russia, and USA. Table
1 shows brief description for the seven climate prediction models. All selected
models have retrospective forecast for the 21-year period of 1983-2003 with 4month integrations initialized in February 1, May 1, August 1, and November 1
targeting one-month lead seasonal prediction. Suppose the forecast was
initialized on February 1, the one-month lead seasonal prediction means the
average of predicted March, April, and May means.
We designed metrics for gauging model’s performance in simulating
monsoon climate and variability. The metrics used for assessing the annual
modes of precipitation include long-term annual mean, the two leading modes
of annual variations, and monsoon domain. Temporal correlation skill was
calculated to verify model’s general performance on interannual variability of
precipitation for four seasons. In addition, Season-reliant EOF (SEOF) analysis
is used as the metrics for gauging model’s performance on interannual
variability of Asian-Australian monsoon (A-AM) precipitation.
1. Mean State Performance
Knowledge of the model’s performance in simulating and forecasting
seasonal mean states is necessary for assessing model capability in predicting
188
seasonal anomalies, especially when predicted variables strongly depend on
seasonal variation such as precipitation. The annual variation of precipitation
can be depicted by a simple objective metric consisting of three major
components: the annual mean and the two leading modes of the annual cycle.
The annual mean here is defined by a 21-year and 12-month mean precipitation.
The first two leading modes of the annual cycle in precipitation were derived
from an EOF analysis of the time series of climatological monthly mean
precipitation (Wang and Ding 2007). We also use the monsoon domain as one
of the metrics. The “monsoon domain” is defined as the regions in which the
annual range exceeds 2mmday-1 and the local summer monsoon precipitation
exceeds 35% of annual rainfall. The annual range of precipitation is measured
by the local summer-minus-winter precipitation, i.e., JJA – DJF for northern
hemisphere and DJF – JJA mean precipitation for the southern hemisphere.
(1) Long-Term Annual Mean Precipitation
Figure 1a and 1b show climatological annual mean precipitation in
observation and one-month lead prediction for the period 1983-2003. Overall,
the MME prediction reproduces the observed features realistically, which to a
large extent include (1) the major oceanic convergence zones over the Tropics,
(2) the major precipitation zones in the extra-tropical Pacific and Atlantic, which
are associated with the oceanic storm tracks, and (3) remarkable longitudinal
and latitudinal asymmetries.
The common biases in simulating annual mean precipitation are seen in
Fig. 1c. The models tend to underestimate precipitation over the eastern Indian
189
Ocean, the equatorial western Pacific, and East Asia Region, but overestimate
precipitation over the Maritime Continent and Philippines where the asymmetric
wind-terrain interaction determines rainfall distribution. The excessive rainfall is
also seen along the Andes, Sierra Madrea, and the southern and eastern flanks
of the Tibetan Plateau where the wind-terrain interaction influences annual
rainfall. Terrain-related precipitation appears more difficult to be captured with
the current climate prediction models because of their coarse resolution and
imperfect parameterization. Precipitation is also overestimated over the central
subtropical Pacific in both hemispheres, because it is associated with the
deficient precipitation over the equatorial western Pacific.
The uncertainties in the MME hindcast are calculated as the standard
deviation of individual model ensemble’s departure from the MME mean. Figure
1d shows that the model spread tends to be large over the region where the
mean biases are found, especially over the Maritime continent and oceanic
convergence zones. In addition, a large spread is found along the high elevated
terrains (e.g., the equatorial section of Andes), indicating that the model
resolution is still one of the major error sources in predicting continental
precipitation.
Figure 2 show the annual mean precipitation derived from CMAP and
individual model prediction. Most of models well capture the observed spatial
pattern of annual mean precipitation but some models, such as M2 and M6
have difficulties in capturing mean precipitation especially over Western Pacific
and South Asia region. M4 has difficulties in capturing the major precipitation
zones in the extra-tropical Pacific and Atlantic.
190
The performance of the individual coupled models and their MME in
predicting annual mean precipitation is evaluated by using a pattern correlation
coefficient (PCC) and a root mean square error normalized (NRMSE) by the
observed standard deviation against an area mean. The MME prediction is in
good agreement with observation in terms of both PCC (0.91) and NRMSE
(0.42) over the global tropics and subtropcis [0-360oE, 40oS-40oN] (Fig. 2). The
best (worst) model has a PCC of 0.93 (0.57) and a NRMSE 0.38 (0.85). It is
also noted that the PCC and NRMSE have a strong linear relationship,
indicating that a model with a higher PCC has a lower NRMSE.
(2) Annual Cycle of Precipitation
Wang and Ding (2007) defined two annual cycle modes by EOF analysis
of the climatological monthly mean precipitation. They have shown that the first
two leading modes account for 71% and 13% of the total annual variance,
respectively. The first mode represents a solstice global monsoon mode. Its
spatial pattern can be represented extremely well by the difference between
Jun-July-August-September (JJAS) and December-January-February-March
(DJFM) mean precipitation. The second EOF mode has also an annual period
with a maximum in April-May and minimum in October-November. The second
mode represents an equinox asymmetric mode and its spatial pattern can be
well represented by the difference between April-May (AM) and OctoberNovember (ON). Hereafter, we will use the pattern of JJAS minus DJFM and AM
minus ON to represent, respectively, the 1st and 2nd annual cycle modes.
It is shown in Fig. 4 that the first two annual cycle modes of the one-month
191
lead MME seasonal prediction are reasonably close to the observed
counterparts. The major deficiencies with the predicted first mode are found
over the Bay of Bengal, South China Sea, Western North Pacific, and East
Asian monsoon front (Maiyu-Baiu-Changma rain band), implying that the MME
predicted a weaker-than-observed Asian summer monsoon (Fig. 4c).
The 2nd annual cycle mode, or the spring-fall asymmetric mode, is
captured realistically by the MME prediction (Fig. 4e) but with less fidelity than
the 1st mode with due regard to their respective amplitudes. The mean bias,
which is the difference between the MME prediction and observation (Fig. 4f),
shows three major features. First, the strengthening of spring-fall asymmetry
over the entire Indian Ocean sector is linked to enhancing (reducing)
precipitation over eastern (western) Indian Ocean in both AM and ON. Second,
the enhancement of spring-fall asymmetry over East Asia and South China SeaWestern North Pacific regions is mainly due to a positive bias of precipitation
over continental East Asia and a negative bias over the oceanic region in AM.
Third, the negative bias of ITCZ results from overestimating (underestimating)
of ITCZ in ON (AM).
Figures 5 and 6 show the spatial pattern of the first and the second annual
cycle mode, respectively, derived from CMAP and individual model ensemble
predictions. The individual models show different systematic errors in predicting
the two annual cycle modes. Especially, the models have difficulties in capturing
the 2nd model. The performance of the individual ME and their MME predictions
in simulating annual cycle is assessed in terms of PCC and NRMSE over the
globe. Figure 7 shows that the current climate models can reproduce the
192
observed 1st annual cycle mode realistically with a similar degree of skill as that
for annual mean (Fig. 3). But the model scatter for NRMSE is higher in the 1st
AC mode (0.57~1.02) than in the annual mean (0.38~0.85). On the other hand,
the models tend to have difficulty in simulating the 2 nd annual cycle mode. They
show large spreading in PCC (0.45~0.75) and NRMSE (0.79~2.02) in Fig. 7b.
However, the MME has evidently a reasonable degree of skill: the PCC is 0.75
and the NRSME is 0.78. It is noted that the linear relationship between the two
skill measurements is weaker for the 2nd mode (Fig. 7b) compared to the 1st
mode (Fig. 7a) and the annual mean (Fig. 3).
(3) Monsoon domain
We also use the monsoon domain as one of the metrics to evaluate
climate model’s capability in simulating global precipitation distribution.
According to Wang and Ding (2006), the “monsoon domain” is defined as the
regions in which the annual range exceeds 2mmday-1 and the local summer
monsoon precipitation exceeds 35% of annual rainfall. The annual range of
precipitation is measured by the local summer-minus-winter precipitation, i.e.,
JJA-DJF for the northern hemisphere and DJF-JJA mean precipitation for the
southern hemisphere.
Figure 8 shows monsoon domain depicted by CMAP (black contour) and
the one-month lead MME prediction (red contour). The prediction can
realistically capture the major monsoon domain in South Asia, IndonesiaAustralia, North and South Africa, and Central and South America. However, the
MME prediction has difficulty in capturing the western North Pacific (WNP)
193
monsoon regions and the East Asian (EA) subtropical Maiyu-Baiu-Changma
region.
In the East Asian subtropical and western North Pacific monsoon regions,
the climate models show a large discrepancy in terms of defining the monsoon
(Fig. 8b). The model spread in these regions is depicted by the number of
models which capture monsoon domain at each grid point. Gray color indicates
that all 7 models capture monsoonal precipitation characteristics at the point.
The deficiency arises from the fact that some models cannot predict correctly
the seasonal distribution of precipitation in the East Asian subtropics and
western North Pacific, nor in the southwest Indian Ocean monsoon regions.
Figure 9 shows how the individual models predict monsoon domain.
2. Performance on Interannual Variability
In order to evaluate the models’ performance on interannual variability of
seasonal mean precipitation, temporal correlation skill was calculated for four
seasons, separately. We demonstrate that the temporal correlation skill is
regarded as general verification method and it has strong linear relationship
with two probabilistic skill measures, Brier Skill Score (BSS) and Area under
ROC curve (Aroc), as well as normalized root mean square error (NRMSE)
which is another skill measure of deterministic forecast. In addition, seasonreliant EOF (SEOF) analysis is used as the metrics for gauging model’s
performance on interannual variability of Asian-Australian monsoon (A-AM)
precipitation.
(1) Temporal Correlation Skill for Seasonal Precipitation
194
Figure 10 shows the performance of the MME prediction system on onemonth lead seasonal prediction in terms of temporal correlation skill for 21 years
from 1983 to 2003. The correlation coefficients which are higher than 0.5 are
generally observed over the tropical Pacific and Atlantic between 10S and 20N
in most seasons. Prediction in DJF, SON and MAM is evidently better than JJA
due to the model’s capacity in capturing the ENSO teleconnections around the
mature phases of ENSO. In JJA, while the skill increases over the North Pacific
and Atlantic due to northward migration of the thermal equator, the skill over the
Indian Ocean and the continental summer monsoon regions are very low. The
correlation skill in the Asian-Australian monsoon (A-AM) region remains
moderate, varying from 0.3 to 0.5 depending on season. In DJF, the correlation
skill is very low over Indian Ocean, South Pacific Ocean, and South America
region. African regions are lacking skills in all seasons. Figures 11-14 show
temporal correlation skill of individual model predictions for four seasons,
separately.
Figure 15a shows the linear relationship between Temporal Correlation
Skill (TCC) and NRMSE for DJF mean precipitation if the TCC is over than 0.4.
Thus, The data examined are DJF forecast of precipitation at each grid over the
global Tropics (0o-360oE, 30oS-30oN) the TCC can represent model’s skill for
interannual variability of seasonal anomalies to some extent. Are forecast skills
of the multi-model probabilistic forecast related to the MME deterministic
forecast? Figures 15b and 15c show general relationship among the
deterministic TCC, and probabilistic BSS and AROC scores. Obviously, their
relationships are nonlinear, but the relationship tends to be linear when the skills
195
are reasonably high, for instance, TCC exceeds 0.6, AROC exceeds 0.7 and
BSS exceeds 0.1.
(2) Dominant modes of A-AM Interannual Variability
The year-to-year variation in the vast A-AM region exhibits enormous
regional differences and depends strongly on the phase of the annual cycle.
Based on this physical consideration, Wang and An (2005) have put forth a
Season-reliant Empirical Orthogonal Function (S-EOF) analysis method to
distinguish modes of variability that evolve with the seasons. Their S-EOF anal
ysis of the Indo-Pacific SST anomalies yielded two statistically significant le
ading modes that are not obtainable by using conventional EOF analysis.
The two leading modes represent Low-Frequency (LF) and Quasi-Biennial
(QB) modes, which are distinguished from each other in their seasonal
evolution, spatial structure of the fractional variance, and interdecadal variation
and trend.
The purpose of the S-EOF is to depict seasonally evolving anomalies
throughout a full calendar year. Here, we adopted the concept of the “monsoon
year” (Yasunari 1991), which spans from the summer (June, July and August) of
Year 0, or “JJA(0),” to the spring (March, April, and May) of the following year
(Year 1), or “MAM(1)”. For this purpose, a covariance matrix was constructed
using four consecutive seasonal mean anomalies for each year; in other words,
the anomalies for JJA(0), SON(0), DJF(0/1), and MAM(1) were treated as a
“yearly block” that is labeled Year 0—the year in which the sequence of
anomalies commences. After the EOF decomposition is finished, the yearly
196
block is then divided into four consecutive seasonal anomalies, to obtain a
seasonally evolving pattern of the monsoon anomalies in each monsoon year
for each eigenvector.
We have applied the S-EOF analysis to both observed and predicted
seasonal mean precipitation anomalies, which are the departures from the
mean annual cycle derived from the period of 1983–2003. In the present study,
we consider the A-AM region as extending from 40oE to 160oE, and from 30oS
to 40oN, which covers South Asia and Australia as well as nearly the entire
Indo-Pacific warm pool region.
Temporal Variation
The S-EOF analysis of CMAP precipitation seasonal anomalies (1983–
2003) yields two statistically distinguished leading modes, which account for 3
0.1% and 13.1% of the total variance in precipitation anomalies, respective
ly. Figure 1 shows the time series of the principal component (PC) of the first
and second S-EOF mode. The PCs of the two leading modes are closely
related to El Niño-Southern Oscillation as measured by the NINO 3.4 SST
anomalies (not shown). Note that the A-AM precipitation seasonal anomaly from
JJA(0) to the next MAM(1) is centered on November and December of Year 0,
thus, the PCs have a yearly resolution centered on November-December of
each year. Wang et al. (2007) showed that the first mode is associated with the
turnabout of warming to cooling in the El Nino-Southern Oscillation (ENSO),
whereas the second mode leads the warming/cooling by about one year,
signaling precursory conditions for ENSO. The MME prediction is reasonably
197
well correlated with observation for the first mode with a correlation skill of 0.94
but has difficulty in capturing the 2nd mode with a correlation skill of 0.59.
Spatial Variation
The MME’s hindcast also faithfully reproduces the major spatial
distributions of the two observed leading modes of the interannual variability of
A-AM seasonal precipitation (Fig. 17). This is important because a realistic
temporal evolution doesn’t warrant the corresponding spatial patterns are
realistic. The current evaluation method relies on the fact that the S-EOF spatial
patterns from the observation and models are quite similar. For the first S-EOF
mode (Fig. 17a), the anomalous patterns from JJA(0) to DJF(0/1) are very well
reproduced, with the anomaly pattern correlation coefficients being over 0.78.
For the second mode (Fig. 17b), the anomaly patterns from DJF(0) to MAM(1)
are reproduced reasonably well with map correlation coefficients. However,
the JJA(0) and SON(0) pattern were poorly predicted.
Comparison of the MME Forecast with Reanalysis
In order to appreciate the success of the MME’s hindcast, it is useful to
compare the climate models’ MME hindcast with the two reanalysis datasets,
ERA-40 and NCEP-2. For this purpose, we used both anomaly pattern
correlation and temporal correlation coefficients to assess the skills for the
spatial pattern (Fig. 18a) and principal component (Fig. 18b) of the first two
leading modes. The skill for CliPAS one-Tier MME prediction is also compared.
In Fig. 18, the CMAP is used as the baseline for comparison. The leading
198
modes derived from the MME predictions are in general better or at least
comparable to those derived from the two reanalysis datasets although
individual model predictions show large spread and lower skill than the two
reanalysis datasets. In terms of the spatial pattern (Fig. 18a), the MME
prediction of the first mode is significantly better than the corresponding one in
the NCEP-2 and slightly better than that in ERA-40; for the second mode, the
MME is than ERA-40 and worse than NCEP-2. In terms of temporal evolution
(Fig. 18b), the APCC MME shows considerably higher temporal correlation
coefficients than the two renalayses for the first leading mode. Note also that
the MME outperforms each individual model’s ensemble. The CliPAS one-tier
prediction is significantly better than two reanalysis datasets and APCC two-tier
MME prediction for all metrics.
Reference
Wang, B. and Q. Ding, 2006: Changes in global monsoon precipitation over the
past 56 years. Geophys. Res. Lett., 33, L06711, doi: 10.1029/2005GL
025347.
Wang, B. and Q. Ding, 2007: The global monsoon: major modes of annual
variation in Tropical precipitation and circulation. Accepted to Dynamics
of Atmos. and Ocean.
Wang, B., J.-Y. Lee, I.-S. Kang, J. Shukla, J.-S. Kug, A. Kumar, J. Schemm, J.J. Luo, T. Yamagata, and C.-K. Park, 2007: How accurately do coupled
climate models predict the leading modes of Asian-Australian monsoon
interannual variability? Clim. Dyn. DOI: 10.1007/s00382-007-0310-5
Wang, B. and S.-I. An, 2005: A method for detecting season-dependent modes
of climate variability: S-EOF analysis. Geophys. Res. Lett. 32:L15710
(doi:10.1029/2005GL022709)
199
Yasunari T., 1991: The monsoon year- A new concept of the climatic year in the
Tropics. BAMS 72: 1331-1338.
Table and Figure Captions
Table 1 Description of models participated in APCC operational MME prediction
Fig. 1. The spatial pattern of long-term annual mean precipitation derived from
(a) CMAP, and (b) ten coupled models’ multi-model ensemble (MME)
prediction. (c) The mean bias defined by the difference between (b) and
(a). (d) Model spread against MME mean defined by the root mean
square difference between MME and individual ME predictions.
The
unit is mmday-1.
Fig. 2. The spatial pattern of long-term annual mean precipitation derived from
(a) CMAP and (b)-(h) individual model predictions.
Fig. 3. Forecast skill for annual mean precipitation in terms of pattern correlation
and normalized RMS error over the globe [0-360E, 40S-40N] in MME
and individual model predictions.
Fig. 4. The Spatial pattern of differential precipitation between (a, b) JuneSeptember and December-Marc (JJAS minus DJFM), and (d, e) between
April-March and October-November (AM minus ON) in observation and
MME prediction. (c, f) Bias of MME prediction against CMAP for each
mode. The unit is mmday-1.
Fig. 5. The Spatial pattern of differential precipitation between June-September
and December-Marc (JJAS minus DJFM), in (a) observation and (b)-(h)
individual model predictions. The unit is mmday-1.
Fig. 6. Same as Fig. 5 except for the difference between AM and ON mean
precipitation.
Fig. 7. Forecast skill for (a) the first and (b) the second annual cycle modes in
terms of pattern correlation and normalized RMS error over the globe [0360E, 40S-40N] in MME and individual model predictions.
Fig. 8. (a) The monsoon domain captured by CMAP (black contour) and the
one-month lead MME prediction (blue contour). (b) The number of model
which captures monsoon domain at each grid point.
200
Fig. 9.
(a) Same as Fig. 8a. (b)-(h) Same as Fig. 8a except for individual
model ensemble prediction.
Fig..10 .Spatial distribution of correlation coefficients between the predicted and
the corresponding observed precipitation for the 21 years of 1983-2003
in (a) MAM, (b) JJA, (c) SON, and (d) DJF using 7 climate prediction
models which participate in APCC operational prediction.
Fig. 11. (a) Same as Fig. 10a. (b)-(h) Same as (a) except for individual model
skill.
Fig. 12. (a) Same as Fig. 10b. (b)-(h) Same as (a) except for individual model
skill.
Fig. 13. (a) Same as Fig. 10c. (b)-(h) Same as (a) except for individual model
skill
Fig. 14. (a) Same as Fig. 10d. (b)-(h) Same as (a) except for individual model
skill.
Fig. 15. Scatter diagram of forecast skils of DJF precipitation between (a) TCC
and NRMSE, (b) TCC and Aroc, (c) TCC and BSS, and (d) Aroc and BSS
at each grid points over the global Tropics
Fig. 16. Principal components of the first (a) and the second (b) S- EOF modes
of seasonal precipitation anomaly obtained from CMAP observation
(solid), MME (red), and each model prediction (dotted), respectively.
Fig. 17 . Spatial patterns of the first and the second S-EOF eigenvector of
seasonal (JJA to MAM) precipitation obtained from CMAP observation
and MME prediction.
Fig. 18. The pattern correlation skill of (a) eigenvectors and the temporal
correlation skill of (b) PC time series of the first two S-EOF modes in
MME, each ensemble prediction, ERA 40 and NCEP-2 reanalysis with
CMAP prediction
201
Table and Figures
Table 1 Description of models participated in APCC operational MME prediction
Model
Institute
Model Name
Resolution
China
NCC
T63L16
M6
Japan
JMA
T63L40
M3
GDAPS/KMA
T106L21
M2
GCPS/SNU
T63L21
M1
METRI/KMA
4ox5o L17
M4
Russia
MGO
T42L14
M5
USA
NCEP CFS
T62L64
M7
Korea
202
Index
Fig. 1. The spatial pattern of long-term annual mean precipitation derived from
(a) CMAP, and (b) ten coupled models’ multi-model ensemble (MME)
prediction. (c) The mean bias defined by the difference between (b) and
(a). (d) Model spread against MME mean defined by the root mean
square difference between MME and individual ME predictions.
unit is mmday-1.
203
The
Fig. 2. The spatial pattern of long-term annual mean precipitation derived from
(a) CMAP and (b)-(h) individual model predictions.
204
Fig. 3. Forecast skill for annual mean precipitation in terms of pattern correlation
and normalized RMS error over the globe [0-360E, 40S-40N] in MME
and individual model predictions.
205
Fig. 4. The Spatial pattern of differential precipitation between (a, b) JuneSeptember and December-Marc (JJAS minus DJFM), and (d, e) between
April-March and October-November (AM minus ON) in observation and
MME prediction. (c, f) Bias of MME prediction against CMAP for each
mode. The unit is mmday-1.
206
Fig. 5. The Spatial pattern of differential precipitation between June-September
and December-Marc (JJAS minus DJFM), in (a) observation and (b)-(h)
individual model predictions. The unit is mmday-1.
207
Fig. 6. Same as Fig. 5 except for the difference between AM and ON mean
precipitation.
208
Fig. 7. Forecast skill for (a) the first and (b) the second annual cycle modes in
terms of pattern correlation and normalized RMS error over the globe [0360E, 40S-40N] in MME and individual model predictions.
209
Fig. 8. (a) The monsoon domain captured by CMAP (black contour) and the
one-month lead MME prediction (blue contour). (b) The number of model
which captures monsoon domain at each grid point.
210
Fig. 9.
(a) Same as Fig. 8a. (b)-(h) Same as Fig. 8a except for individual
model ensemble prediction.
211
Fig..10 .Spatial distribution of correlation coefficients between the predicted and
the corresponding observed precipitation for the 21 years of 1983-2003
in (a) MAM, (b) JJA, (c) SON, and (d) DJF using 7 climate prediction
models which participate in APCC operational prediction.
212
Fig. 11. (a) Same as Fig. 10a. (b)-(h) Same as (a) except for individual model
skill
213
Fig. 12. (a) Same as Fig. 10b. (b)-(h) Same as (a) except for individual model
skill.
214
Fig. 13. (a) Same as Fig. 10c. (b)-(h) Same as (a) except for individual model
skill.
215
Fig. 14. (a) Same as Fig. 10d. (b)-(h) Same as (a) except for individual model
skill.
216
Fig. 15. Scatter diagram of forecast skils of DJF precipitation between (a) TCC
and NRMSE, (b) TCC and Aroc, (c) TCC and BSS, and (d) Aroc and BSS
at each grid points over the global Tropics
217
Fig. 16. Principal components of the first (a) and the second (b) S- EOF modes
of seasonal precipitation anomaly obtained from CMAP observation
(solid), MME (red), and each model prediction (dotted), respectively.
218
Fig. 17. Spatial patterns of the first and the second S-EOF eigenvector of
seasonal (JJA to MAM) precipitation obtained from CMAP observation
and MME prediction.
219
Fig 18. The pattern correlation skill of (a) eigenvectors and the temporal
correlation skill of (b) PC time series of the first two S-EOF modes in
MME, each ensemble prediction, ERA 40 and NCEP-2 reanalysis with
CMAP prediction
220