Regional Climate Model Evaluation System (RCMES):
Combining Observations & IT to Establish Core Climate Model
Assessment Capabilities
Jinwon Kim and Paul Ramirez
and
RCMES Science and IT Teams led by
Duane Waliser (JPL), Science Leader
Chris Mattmann (JPL), IT Leader
NCPP Workshop, Boulder, Colorado, August 2013
RCMES: Combining Observations & IT to Establish Core Climate Model Assessment Capabilities
•
Model evaluation is a key step in assessing climate model fidelity and
in turn the (un)certainty of climate change impacts.
•
Systematic experimentation and evaluation of GCMs have been
undertaken for some time (e.g., AMIP, CMIP, CFMIP), while that for
RCMs (e.g., NARCCAP, CORDEX) being more recent and less mature.
•
NASA can provide critical and unique observational data to facilitate
RCM evaluations and thus make key contributions to impact assessment
process, e.g., the National Climate Assessment (NCA).
•
RCMES was developed to facilitate model evaluations via easy access
to observational data, especially from satellite remote sensing, and
software tools for calculating and visualizing evaluation metrics.
RCMES: Combining Observations & IT to Establish Core Climate Model Assessment Capabilities
(http://rcmes.jpl.nasa.gov; Powered by Apache Open Climate Workbench)
Other Data Centers
(ESGF, DAAC, ExArch Network)
URL
Metadata
TRMM
Extractor
for
various
data
formats
AIRS
CERES
Soil
moisture
Data Table
Raw Data
Extract model
data
Regridder
(Put the OBS & model data on
the same time/space grid)
Cloud
Database
Data Table
Data extractor
(Binary or netCDF)
Data Table
Data Table
Data Table
Analyzer
Visualizer
(Plot the metrics)
RCMED
RCMET
(Regional Climate Model Evaluation Database)
(Regional Climate Model Evaluation Tool)
RCMES is flexible and open source software
•
•
•
Users' own
software for
specific
evaluation
and analysis
Calculate evaluation metrics &
assessment model input data
Common Format,
Native grid,
Efficient architecture
ETC
•
Extract OBS
data
Data Table
MODIS
Model data
User
input
Can utilize multiple (distributed) data sources.
Easily transferrable to local platforms (Linux, Mac-OS)
Open source (via Apache)
Assessment
modeling
RCMED Datasets: Satellite retrievals, Surface analysis, Reanalysis, Assimilations
*
*
*
*
*
*
*
*
*
*
*
MODIS (satellite cloud fraction): [daily 2000 – 2010]
TRMM (satellite precipitation): 3B42 [daily 1998– 2010]
AIRS (satellite surface + T & q profiles) [daily 2002 – 2010]
CERES and GEWEX-SRB radiation – Surface and Top of the atmosphere
NCEP CPC Rain Gauge analysis (gridded precipitation): [daily 1948 – 2010]
CRU TS 3.1: precipitation, Tavg, Tmax, Tmin [monthly means, 1901 – 2006]
University of Delaware precipitation and temperature analysis
Snow Water Equivalent over Sierra Nevada Mts [monthly 2000-2010]
NASA MERRA Land Surface Assimilation & pressure-level data [daily, 1979-2011]
ERA-Interim (reanalysis): [daily 1989 – 2010]
AVISO sea-level height [1992-2010]
* (In progress) CloudSat atmospheric ice and liquid, Satellite-based snow (Himalayas),
ISCCP cloud fraction, Fine-scale SST, etc.
RCMET Metrics:
*
*
*
*
*
*
*
•
Bias (e.g. seasonal means or variance)
RMS error (e.g. interannual variability)
Anomaly Correlation (spatial patterns of variability)
PDFs (likelihoods, extremes and their changes)
Taylor Plots & Portrait Diagrams (overall model performance)
Statistical Tests
User-defined regions (e.g. water shed, desert, sea, political)
Datasets and metrics are continuously updated.
RCMES: Combining Observations & IT to Establish Core Climate Model Assessment Capabilities
RCMES is being (will be) used for model evaluation in a number of regional
climate experiment regions via worldwide collaboration
• N. America – NARCCAP
• WCRP CORDEX Regions
• Ongoing/tested: Africa, South Asia, Middle East – N. Africa, Australia
• Under arrangement: Caribbean, South America, East Asia, Arctic
Not Illustrated Here
Arctic & Antarctic Domains
NARCCAP Multi-decadal Hindcast Evaluation: Surface Insolation
Kim et al., 2013, J. Climate, 26,
5698-5715.
Considerable biases with
large-scale spatial structures
exist in surface insolation
fields.
Figure. Surface insolation biases against the GEWEX SRB data
Interannual variability (% SRB)
Bias (% SRB)
Model performance varies
according to regions and
seasons.
Figure. Evaluation of the mean & interannual variability
in various regions within the conterminous US.
Model Evaluation and Observational Uncertainties
RCMES' capability to handle multiple model- and/or/both observation datasets
allows to visualize model performance as well as uncertainties in the model
evaluation due to observational data.
Observational Uncertainty
TRMM, CPC, CRU, UDEL, GPCP
Figure. Evaluation of the annual-mean
climatology from multiple RCMs and
observations against the equal-weight multiobservation ensemble. The figure shows that
there exist noticeable uncertainties among
widely used observations.
Kim et al., 2013, J. Climate, 26, 5698-5715.
Construct watershed-mean met data for hydrology model
* Transferring the gridded climate model data onto a watershed is the first step
in assessing water resources using a bulk hydrology model that runs on
watershed-mean met data.
1.
2.
Overlay model grid over the watershed area.
3.
Calculate the watershed-mean value of a variable P using the weights
Calculate the percentage of each grid box contained within the watershed area.
This is the weighting factor for calculating the area-mean meteorological data.
The Sacramento River
basin
Area
mapping
Map the watershed area onto the
RCM domain (Kim et al. 2000, J.
Hydromet.)
The shaded area is the 0.5oresoln RCM grid boxes that are
entirely or partially included in
the basin.
Figure. Calculation of area-mean data for an irregularly-shaped watershed from gridded climate
model data.
Evaluation of extreme events
Analysis of long-tailed temperature PDFs in NARR and NARCCAP RCMs:
•
Variance/skewness of daily temperature are related with weather extremes.
•
These PDF properties can also be related with regional processes
•
•
e.g., Positive tail in the LA basin is related with the Santa Ana winds
Utilize these PDF characteristics to evaluate RCMs in simulating extreme events
Loikith et al., 2013: Geophys. Res. Lett., in press.
Regional Evaluation of GCMs
* RCMES can be also used to evaluate multiple GCMs for various regions using
multiple observation data.
* Can be useful, for example, in identifying GCMs most suitable for downscaling.
CMIP5 GCMs - Observation
NARCCAP RCMs - Observation
Fig. July precipitation bias: CMIP5 present-day Ens & NARCCAP RCM Ens
Summary
•
The Regional Climate Model Evaluation System (RCMES) was developed to
facilitate model evaluation via easy access to key observational datasets
especially from satellite remote sensing data sets.
•
RCMES is being used in evaluation studies for several regional climate
experiments including NARCCAP and multiple CORDEX domains.
•
•
RCMES includes calculation of a number of metrics for model evaluation.
•
RCMES is useful for a wide range of users:
RCMES is continuously improved with additional datasets, more efficient
database scheme, and evaluation metrics.
•
•
•
Can be obtained and installed as a Virtual Machine image
•
http://rcmes.jpl.nasa.gov/training/downloads
Deployable to user laptops and server machines with ease
Source Code at Apache Open Climate Workbench
•
•
•
•
http://climate.incubator.apache.org/
Command line interface
Browser based interface
API to construct your own evaluations
Ongoing and Future Efforts
•
Links to ESGF
•
•
Made-to-order system
•
•
•
Distribution of RCMES in a virtual machine package as specified by users.
Link climate model data with assessment models
•
Preparation of met forcing data from climate model data, bias correction of
met forcing data for specific assessment models.
•
Performance-based variable weighting model ensemble construction
Data processing and metrics
•
•
•
•
Enable users to import data directly from ESGF for processing.
Handling of very large datasets.
Develop metrics for evaluating PDF characteristics (e.g., variance, skewness)
Cluster analysis
Collaborative Development through Apache Open Climate Workbench
•
Open source effort based on a well known open source software community
and team expertise at Apache
RCMES: Evaluation of Multi-model Hindcast in the CORDEX Africa
jkim@atmos.ucla.edu; duane.e.waliser@jpl.nasa.gov; chris.a.mattmann@jpl.nasa.gov
Terrain of the evaluation domain and the annual precipitation climatology
Evaluation of the spatial variability of the annual mean precipitation
Evaluation of the simulated precipitation annual cycle
RMSE (% Annual mean)
Correlation coefficients
Kim, J., D. Waliser, C. Mattmann, C. Goodale, A. Hart, P. Zimdars, D. Crichton, C. Jones, G. Nikulin, B. Hewitson, C. Jack, C. Lennard, and A. Favre,
2013: Evaluation of the CORDEX-Africa multi-RCM hindcast: systematic model errors. Clim. Dyns, DOI 10.1007/s00382-013-1751-7.
Project supported by:
NASA: NCA (11-NCA11-0028), AIST (AIST-QRS-12-0002), American Recovery and Reinvestment Act.
NSF: ExArch (1125798), EaSM (2011-67004-30224)
RCMES: Precipitation Evaluation of Multi-model Hindcast in the CORDEX South Asia – Indian subcontinent
The hindcast and evaluation domain
Model biases in simulating spatial variations in the annual-mean climatology
Hindcast domain
RCMs
Indian subdomain
Observations
mm/day
Uncertainties in the Observations
Annual cycle evaluation for the northern mountainous region
R01
R01
R02
R03
R04
R04
R05
R06
% of Observation Ensemble
Supported by: NASA: NCA (11-NCA11-0028), AIST (AIST-QRS-12-0002); NSF: ExArch (1125798)
RCMES – Regional Climate Model Evaluation System
http://rcmes.jpl.nasa.gov/
Global Climate
Projections
Sponsor Acknowledgements
•
NSF G8 Initiative: ExArch Project
•
NASA: National Climate Assessment (NCA)
•
NASA: Adv. Information. Systems Technology (AIST)
•
NASA: Comp. Modeling Alg. & Cyberinfras. (CMAC)
Selected Publications
•
•
Crichton et al., IEEE Software, 29, 63-71.
•
•
Kim et al., 2013a, J. Climate, 26, 5698-5715.
•
•
Loikith et al.,2013, Geophys. Res. Lett., in press.
•
Waliser et al., 2012, Tech. input for consideration to the
2013 Us National Climate Assessment, 19pp.
•
Whitehall et al., 2012, WMO Bulletin, 61, 29-34.
Hart et al., ICSE 2011 Workshop Software Engineering for
Cloud Computing - SECLOUD, Honolulu, HI, May 2011, 43-49,
ISBN: 978-1-4503-0582-2, doi: 10.1145/1985500.1985508.
Kim et al., 2013b, Climate Dynamics, DOI 10.1007/s00382013-1751-7.
Mattmann et al., 2013, Earth Sci. Informatics, doi
10.1007/s12145-013-0126-2.
plus a number of IT papers and conference papers
Regional
Downscaling
Decision
Support