QPF statistical properties
and verification
of COSMO models over Italy
Maria Stefania Tesini
Carlo Cacciamani & Tiziana Paccagnella
11th COSMO General Meeting
7-11 September 2009, Offenbach
Motivation




Observed precipitation fields show a high variability both in space
and time and the amount of rainfall could vary a lot within a short
distance
High Resolution models seem to be able to reproduce this
variability, even if frequent errors in time and space positioning
make difficult a grid-point based employment of models QPF
In order to appreciate the properties and the additional information
provided by LAMs respect to coarser resolution models we
developed a strategy based on the aggregation of the observed
values and forecasted values that fall within a predefined
geographical area
In this work we present various approaches to QPF validation,
focusing our attention on the behaviour of the COSMO
implementations that run operationally at ARPA-SIMC (COSMO-I7
and COSMO-I2), in comparison with the global model IFS-ECMWF
Maria Stefania Tesini
COSMO MEETING 2009
Dataset
24 hours accumulated precipitation
(00-24 UTC or 6-6 UTC)

OBSERVATION:
– More than 1000 rain-gauges provided by
» Italian Department of Civil Protection
network
» COSMO dataset
» Meteo Swiss (SON2008)
 Non homogeneous dataset both in space
and time

MODELS:
– 00 UTC run
– +24h or +6h to +30h forecast
– COSMO-I7 ~ 7 km h.r.
– COSMO-I2 ~ 2.8 km h.r.
– IFS-ECMWF ~ 25 km h.r. *
* run 12 UTC (+12h to +36h) and ~ 50 km h.r. before 2006
Maria Stefania Tesini
COSMO MEETING 2009
Methodology


The domain is divided in squared areas (boxes) of chosen size
The values of all stations and all model grid-points that fall in the
same box are aggregated and processed essentially using two
different approach
Evaluation of a summarizing
value for the precipitation field
in each box:
- mean
- maximum
- 90 th percentile
- ….
Study of the
distribution function of
the precipitation filed in
each box
Maria Stefania Tesini
COSMO MEETING 2009
Quality of the forecast: Verification Scores

Size of the boxes: 0.5° x 0.5°
– Selected in order to contains 4 ECMWF grid-points
– variable number of grid-points for COSMO models
» COSMO-I7 ~ 40 – 45 grid-points
» COSMO-I2 ~ 270 - 290 grid-points
– Each box should contain at least 3 stations points to be considered in the
computation

Definition of the event:
the mean value (or the maximum value) in the box exceed a threshold
» 1 mm/24h - 5 mm/24h - 10 mm/24h - 20 mm/24h


Creation of the contingency table
Evaluation of verification scores
Maria Stefania Tesini
COSMO MEETING 2009
SON 2008 - Mean precipitation > 1mm/24h
COSMO-I2
Maria Stefania Tesini
COSMO-I7
COSMO MEETING 2009
ECMWF
SON 2008 - Mean precipitation > 20 mm/24h
COSMO-I2
Maria Stefania Tesini
COSMO-I7
COSMO MEETING 2009
ECMWF
SON 2008 - Max precipitation > 20 mm/24h
COSMO-I2
Maria Stefania Tesini
COSMO-I7
COSMO MEETING 2009
ECMWF
Distribution function of precipitation




Assessment of observed and
forecasted precipitation “climatology”
over well-defined areas by the study
of the distribution function (pdf)
Estimation of the pdf of
observed/predicted precipitation field
taking in account all the points that
fall within each area throughout the
examined periods
The size of the boxes is 1°x1° in
order to have enough forecast and
observation points to perform
significant statistics
The boxes have been selected on
the basis of orographic features so
that we assume that the precipitation
in each of these points is roughly
equi-probable
Maria Stefania Tesini
COSMO MEETING 2009
Observations: Autumns 2005 & 2007
North-West
North Appenine
N-East
Centre of Italy
Sout of Italy
Sardinia
99 th perc
90 th perc
Maria Stefania Tesini
COSMO MEETING 2009
COSMO-I7: Autumns 2005 & 2007
North-West
Maria Stefania Tesini
North Appenine
N-East
Centre of Italy
COSMO MEETING 2009
Sout of Italy
Sardinia
ECMWF: Autumns 2005 & 2007
North-West
Maria Stefania Tesini
North Appenine
N-East
Centre of Italy
COSMO MEETING 2009
Sout of Italy
Sardinia
Amount of precipitation in a season



Evaluation of the daily mean (max) in each box for
models and observations
Accumulation of the daily mean (max) over the
“common” days
Domain divided in a grid of boxes of different size, in
this case
– 0.5° x 0.5°
» required at least 3 stations in each box
» ECMWF 4 points
» COSMO-I7 ~ 40 – 45 points
» COSMO-I2 ~ 270 - 290 points
Maria Stefania Tesini
COSMO MEETING 2009
SON 2008 - Seasonal sum of daily (6-6UTC) MEAN precipitation
COSMO-I2
ECMWF
Maria Stefania Tesini
COSMO-I7
Box size
0.5° x 0.5°
OBS
COSMO MEETING 2009
SON 2008 - Seasonal sum of daily (6-6UTC) MAX precipitation
COSMO-I2
ECMWF
Maria Stefania Tesini
COSMO-I7
Box size
0.5° x 0.5°
OBS
COSMO MEETING 2009
MAM 2009 - Seasonal sum of daily (00-24 UTC) MEAN precipitation
COSMO-I2
COSMO-I7
Box size
0.5° x 0.5°
ECMWF
Maria Stefania Tesini
OBS
COSMO MEETING 2009
MAM 2009 - Seasonal sum of daily (00-24 UTC) MAX precipitation
COSMO-I2
COSMO-I7
Box size
0.5° x 0.5°
ECMWF
Maria Stefania Tesini
OBS
COSMO MEETING 2009
Variability of daily mean and max during a season
Low quantity for more days
Same sum
achieved in
different
ways


Few events of high rain
In order to appreciate the ability of the models in
representing the variability of the precipitation field
during the period of interest, we studied the distribution
function of the daily mean and of the daily maximum
Size of the chosen areas: 0.5°x0.5°
Maria Stefania Tesini
COSMO MEETING 2009
SON2008 – Seasonal Distribution of daily mean and max
MEAN
OBS
COSMO COSMO
ECMWF
I7
I2
Maria Stefania Tesini
MAX
OBS
COSMO MEETING 2009
COSMO COSMO
ECMWF
I7
I2
SON2008 – Seasonal Distribution of daily mean and max
MEAN
OBS
COSMO COSMO
ECMWF
I7
I2
Maria Stefania Tesini
MAX
OBS
COSMO MEETING 2009
COSMO COSMO
ECMWF
I7
I2
SON 2008 -Seasonal Distribution of daily mean and max
MEAN
OBS
COSMO COSMO
ECMWF
I7
I2
Maria Stefania Tesini
MAX
OBS
COSMO MEETING 2009
COSMO COSMO
ECMWF
I7
I2
SON 2008 -Seasonal Distribution of daily mean and max
MEAN
OBS
COSMO COSMO
ECMWF
I7
I2
Maria Stefania Tesini
MAX
OBS
COSMO MEETING 2009
COSMO COSMO
ECMWF
I7
I2
Day-by-day behaviour: qualitative description



Investigation of models ability in reproducing
the day-by-day variability of the precipitation
field inside a selected area
The precipitation values of all model gridpoints and of all rain-gauges that fall within the
chosen area are taken into account to evaluate
the respective distribution functions
Size of the selected areas: 0.5°x 0.5°
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
ECMWF
COSMO-I7
COSMO-I2
OBS
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
ECMWF
COSMO-I7
COSMO-I2
OBS
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
ECMWF
COSMO-I7
COSMO-I2
OBS
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
ECMWF
COSMO-I7
COSMO-I2
OBS
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
Maria Stefania Tesini
COSMO MEETING 2009
APRIL 2009 - Daily distribution
Maria Stefania Tesini
COSMO MEETING 2009
Amount of precipitation in a season (2)



Evaluation of the daily mean (max) in each box for
models and observations
Accumulation of the daily mean (max) over “common”
days
grid of boxes of size
– 0.25° x 0.25°
»
»
»
»

required at least 1 station in each box
ECMWF 1 points
COSMO-I7 ~ 9 – 12 points
COSMO-I2 ~ 50 - 60 points
Useful tools to compare models
Maria Stefania Tesini
COSMO MEETING 2009
MAM 2008 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
MAM 2008 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
MAM 2008 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
MAM 2008 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
SON 2008 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
SON 2008 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
SON 2008 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
SON 2008 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
MAM 2009 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
MAM 2009 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
MAM 2009 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
MAM 2009 – Models comparison
Maria Stefania Tesini
COSMO MEETING 2009
Conclusion



We approached the validation of models QPF using
mostly a descriptive approach
The non-homogeneous dataset make difficult
summarizing the results
However the methodologies used had pointed out
additional value of High Resolution models:
– Improvements are more evident when the maximum value of
precipitation in a box is investigated (e.g. high localized events)
– The distribution functions (both on a seasonal or daily basis) of
the COSMO models seem to be more realistic than that of
ECMWF
– The features of the pdf of observation , and in particular the
spread of the tail of the distribution, seem to be reasonably well
reproduced by the COSMO models (maybe COSMO-I2 better
than COSMO-I7)
– On the other hand, the spread of ECMWF pdf do not cover all
the range of the observed values
Maria Stefania Tesini
COSMO MEETING 2009
Conclusion

The comparison of the seasonal amount of rain
for the three models had pointes out significant
differences in the structure of the pattern of the
precipitation fields
– Increasing resolution add more orographics details
– COSMO-I2 seemed to be more “dry” in some region
and more “wet” in other (dry upwind and wet
downwind ….to be investigated)
Maria Stefania Tesini
COSMO MEETING 2009
Thanks for your attention
Maria Stefania Tesini
COSMO MEETING 2009