NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Update on the Earth System
Modeling Framework
GFDL FMS Suite
MITgcm
NCAR/LANL CCSM
NCEP Forecast
NSIPP Seasonal Forecast
Cecelia DeLuca, NCAR
CCSM Software Engineering Working Group Mtg
Boulder, CO
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Technological Trends
In climate research and NWP...
increased emphasis on detailed representation of individual
physical processes; requires many teams of specialists to
contribute components to an overall modeling system.
In computing technology...
increase in hardware and software complexity in highperformance computing, as we shift toward the use of scalable
computing architectures.
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Project Overview
GOAL: To increase software reuse, interoperability, ease of use and performance
portability in climate, weather, and data assimilation applications
PRODUCTS:
•
Coupling superstructure and utility infrastructure software
•
Synthetic code suite for validation and demonstration
•
Set of 15 ESMF-compliant applications (including CCSM, WRF, GFDL models;
MIT, NCEP and NASA data assimilation systems)
•
Set of 8 interoperability experiments
RESOURCES: $10.1M over 3 years from NASA Earth Science Technology Office
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Collaborators
NSF NCAR
Tim Killeen, PI
Byron Boville
Cecelia DeLuca
Roberta Johnson
John Michalakes
Al Kellie
Jeff Anderson
David Neckels
Nancy Collins
Jon Wolfe
Earl Schwab
NASA DAO
Arlindo da Silva, PI
Leonid Zaslavsky
Will Sawyer
Carlos Cruz
4 February 2003
NOAA NCEP
Stephen Lord
Mark Iredell
Mike Young
Wei Yu
John Derber
University of Michigan
Quentin Stout
MIT
John Marshall, PI
Chris Hill
DOE Los Alamos National Lab
Phil Jones
NOAA GFDL
Ants Leetmaa
V. Balaji
Robert Hallberg
NASA NSIPP
Max Suarez
Michele Rienecker
Christian Keppenne
Atanas Trayanov
DOE ANL
Rob Jacob
Jay Larson
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Interoperability
Demonstrations
COUPLED CONFIGURATION
NEW SCIENCE ENABLED
GFDL B-grid atm / MITgcm ocn
Global biogeochemistry (CO2, O2), SI timescales.
GFDL MOM4 / NCEP forecast
NCEP seasonal forecasting system.
NSIPP ocean / LANL CICE
Sea ice model for extension of SI system to centennial time scales.
NSIPP atm / DAO analysis
Assimilated initial state for SI.
DAO analysis / NCEP model
Intercomparison of systems for NASA/NOAA joint center for satellite
data assimilation.
DAO CAM-fv / NCEP analysis
Intercomparison of systems for NASA/NOAA joint center for satellite
data assimilation.
NCAR CAM Eul / MITgcm ocn
Improved climate predictive capability: climate sensitivity to large
component interchange, optimized initial conditions.
NCEP WRF / GFDL MOM4
Development of hurricane prediction capability.
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Hierarchy of Modeling
Infrastructure Projects
CCSM Coupled System,
NASA GEMS, GFDL FMS
Earth System
Modeling Framework
Space Weather (SWMF), Geophysics,
Ecosystem Dynamics, …
Common Component Architecture
4 February 2003
Single Earth system
model or modeling system
Climate, Weather, Data
Assimilation
All Earth, Space, and Life
Applications
All HPC Applications
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Related
Projects
PRISM
WRF
CCA
ESMF
FMS
4 February 2003
SWMF
SciDAC
ESMF Earth System Modeling Framework
CCA DOE Common Component Architecture
SciDAC DOE/NSF CCSM SciDAC Project
GEMS Goddard Earth Modeling System
FMS GFDL Flexible Modeling System
SWMF Space Weather Modeling Framework
WRF Weather Research and Forecast Model
CCSM Community Climate System Model
PRISM Program for Int. Earth System Modeling
CCSM
GEMS
Larson/ANL
DeLuca/NCAR-SCD
Jones/LANL
Stout/U Mich
Killeen/NCAR
Drake/ORNL
Boville/NCAR-CGD
Michalakes/NCAR-MMM
Suarez/NASA Goddard
Balaji/NOAA GFDL
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF and CCA
• Common Component Architecture (CCA) is creating a minimal interface
and sets of tools for linking high performance components
• CCA can be used to implement frameworks and standards developed in
specific domains (such as ESMF).
• DOE funded through SciDAC
• Collaborators include LANL, ANL, LLNL, ORNL, Sandia, University of
Tennessee, and many more
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF and CCA, cont.
• Shujia Zhou / Arlindo da Silva
(NASA GSFC) currently
prototyping an ESMF API using
CCA tool CCAFFEINE; working
towards CAM/DAO PSAS
coupling using ESMF and CCA
• Ongoing ESMF collaboration
with CCA/LANL on language
interoperability
• Joint CCA / ESMF COMTESS
meeting January 2003
• Nancy Collins (NCAR SCD) is
ESMF liaison to CCA
4 February 2003
Prototype ESMF / CCA interface in CCAFFEINE display
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF and PRISM
• A European Earth system modeling infrastructure
project, started December 2001
• Funded by the European Commission ($4.8M)
• Involves current state-of-the-art atmosphere,
ocean, sea-ice, atmospheric chemistry, landsurface and ocean-biogeochemistry models
• 22 partners: leading climate researchers and
computer vendors, includes MPI, KNMI, UK Met
Office, CERFACS, ECMWF, DMI.
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF and PRISM, cont.
• Working together to supplement
CF convention for physical field
names and quantities
• Component interface database
developed by ESMF stores fields
in model import and export states
• V. Balaji (GFDL) is ESMF liaison to
PRISM
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Architecture
Coupling Layer
ESMF Superstructure
Model Layer
User Code
Fields and Grids Layer
ESMF Infrastructure
Low Level Utilities
External Libraries
4 February 2003
BLAS, MPI, NetCDF, …
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Components
1.
ESMF provides an environment for assembling geophysical components
into an application.
Application Component
Gridded Components
2.
ESMF provides a toolkit that
components use to
i. increase interoperability
ii. improve performance
portability
iii. abstract common services
4 February 2003
Coupler Components
Component: Init(), Run(), Checkpoint()
Field: Halo(),
Import(), Export() +
I/O
Grid: Regrid(),
Transpose() + Metrics
Layout, PE List, Machine Model
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
General Features
• ESMF will be usable by models written in F90/C/C++
• ESMF will be usable by models requiring adjoint capability
• ESMF will be usable by models requiring shared or distributed memory
parallelism semantics
• ESMF will support SPMD and MPMD coupling
• ESMF will support several I/O formats, including GRIB/BUFR, netCDF,
HDF
• ESMF will have uniform syntax across platforms
• ESMF will target a broad range of platforms, from major centers 
commodity hardware
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Design Principles:
Scalable Applications
Since each ESMF application is also a component, entire ESMF
applications may be treated as gridded components and nested
within larger applications.
climate_comp
ocn2atm_coupler
ocn_comp
atm_comp
phys2dyn_coupler
atm_phys
PE
4 February 2003
atm_dyn
Example: atmospheric
application containing multiple
coupled components within a
larger climate application
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Design Principles:
Local Communication
All inter-component communication within ESMF is local.
climate_comp
This means:
atm2ocn _coupler
ocn_comp
atm_comp
phys2dyn_coupler
atm_phys
PE
4 February 2003
atm_dyn
Coupler components must be
defined on the union of the PEs of
all the components that they
couple.
In this example, in order to send
data from the ocean component to
the atmosphere, the coupler
mediates the send.
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Design Principles:
Modularity
Gridded Components don’t have access to the internals of other Gridded
Components. Gridded Components may
- pass their States through their argument list or
- receive methods, through their argument list, for interacting with other
Components (Transforms).
coupler
transform
call ESMF_CompRun(atm, xform)
call ESMF_CompRun(ocn, xform)
ocn_component
atm_component
call ESMF_StateXform(ex_state, &
xform)
call ESMF_StateXform(xform, &
im_state)
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Design Principles:
Uniform Communication API
The same programming interface is used for shared memory, distributed memory,
and combinations of both.
Machine model = abstraction of machine architecture (num_nodes,
num_pes_per_node)
DE = a decomposition element - may be virtual, thread or MPI process
Layout = an arrangement of DEs, in which dimensions requiring faster
communication may be specified and resources arranged accordingly.
A 2 x 6 Layout of 4 3-processor nodes
4 February 2003
The data in a Grid is decomposed
into the number of chunks specified
in the Layout.
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Class Structure
Component
State
Superstructure
Infrastructure
Bundle
Regrid
Field
Grid
PhysGrid
To do
More work
Prototype exists
DistGrid
Layout
MachineModel
F90
Array
Comm
Utilities: TimeMgr, LogErr, I/O etc.
4 February 2003
Data Communications
Route
C++
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Timeline
May 2002
Draft Developer’s Guide and Requirements Document completed
Community Requirements Meeting and review held in D.C.
July 2002
ESMF VAlidation (EVA) suite assembled
August 2002
Architecture Document: major classes and their relationships
Implementation Report: language strategy and programming
model
Software Build and Test Plan: sequencing and validation
May 12-15 2003 First API and software release, Community Meeting at GFDL
July 2003
First 3 interoperability experiments completed
April 2004
Second API and software release, Community Meeting
July 2004
All interoperability experiments completed
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Development Status
• ~8 FTEs, half in NCAR SCD, actively developing code
• System tests started in early January 2003
• Newly updated ESMF Architecture Document, ESMF Requirements
Document
• Build and Test plans collected from all collaborators as well as core team
• Continually updated Developer’s Guide with coding conventions,
configuration management, QA procedures, more
• Pseudocode application-level interfaces developed by CCSM, NSIPP, NCEP,
NASA DAO
• Documents, system tests, pseudocodes on-line, on ESMF website
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
May 2003 Release
Focus for May 2003 ESMF release is on developing
sufficient infrastructure and superstructure to achieve the
initial set of interoperability experiments.
These are:
– FMS B-grid atmosphere coupled to MITgcm ocean
– CAM atmosphere coupled to NCEP analysis
– NSIPP atmosphere coupled to DAO analysis
4 February 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
More information
ESMF website: http://www.esmf.ucar.edu
On the website:
• ESMF documents
• ESMF browsable source code repository
• On-line forms for community input and catalogued responses
• Milestone schedule, project archives and history
• Related projects
• Management plan, board and team contacts
• More …
4 February 2003