Coupling Climate and Hydrological
Models
Interoperability Through Web Services
Outline
• Project Objective
• Motivation
• System Description
– Components
– Frameworks
– System Driver
•
•
•
•
Logical Workflow
Data Flow
Architecture
Future Directions
Project Objective
The development of a two-way coupled, distributed, servicebased modeling system comprised of an atmospheric climate
model using a standard ESMF interface and a hydrological
model that utilizes standard interfaces from that domain.
Motivation and
Background
• Hydrological impact studies can be improved when forced
with data from climate models; hydrological feedbacks can
affect climate
• A technology and scale gap exists:
– Many hydrological models have limited scalability, run on desktop
computers, and have watershed-sized domains
– Many climate models are highly parallel, run on high performance
supercomputers and have global domains
– However, scales are slowly converging (e.g. high resolution climate
models, hydrological systems of greater extent), and this provides
opportunities to explore new coupled model configurations and
modes of coupling
Approach
• Leveraging frameworks (ESMF, OpenMI) that can
operate as web services is a way to implement loose
coupling between such systems – this has many
advantages
– Loose coupling can be implemented with minimal source code
changes needed to run the models
– Standard connectors between various frameworks and
interfaces can be written, encouraging broad interoperability
– A web service approach enables model components to be
integrated with a heterogeneous network of other services
(data retrieval and post-processing, etc.)
– The web service approach is well understood
– The approach leverages substantial investments in converting
operational weather and water models to ESMF
System Description
• SWAT (hydrology model) runs on a PC
Personal Computer
• CAM (climate model) runs on a high
performance computer
• Wrappers for both SWAT and CAM
provide an OpenMI interface to each
model
Driver
OpenMI
SWAT
CAM OpenMI
Wrapper
• Driver (OpenMI Configuration Editor)
uses OpenMI interface to timestep
through models via wrappers
• The CAM run is invoked
through ESMF web
services
• CAM output data is
streamed to a CAM
wrapper via ESMF web
services
ESMF Web
Services
ESMF CAM
Component
High Performance Computer
Components: SWAT
• The hydrological model
chosen for this project is the
Soil Water Assessment Tool
(SWAT)
• It is a river basin scale model
developed to quantify the
impact of land management
practices in large, complex
watersheds
• It was chosen for this project
because it is widely used,
open source, and has a
standard interface (OpenMI)
Components: CAM
• The atmospheric model chosen for this system is the
Community Atmospheric Model (CAM5), part of the
Community Climate System Model (CESM1.0.3)
• It was chosen because it
is widely used, open
source, and has a
standard interface
(ESMF)
Frameworks: Earth System
Modeling Framework
• ESMF is a high-performance, flexible, operational-quality
software infrastructure that increases the ease of use,
performance portability, interoperability, and reuse in Earth
science applications
• It provides an architecture for composing complex, coupled
modeling systems and can support fully unstructured,
logically rectangular, and observational data structures
• It is highly portable (24+ platforms), highly scalable (tested
to 16K+ processors), and includes 4500+ tests and examples
• Web services included in the ESMF distribution allow any
networked ESMF component to be available as a web
service.
Frameworks: OpenMI
• The OpenMI Software Development Kit (SDK) is a software
library that provides a standardized interface that focuses
on time dependent data transfer
• Primarily designed to work with systems that run
simultaneously, but in a single-threaded environment
[Gregerson et al., 2007]
• The primary data structure in OpenMI is the ExchangeItem,
which comes in the form of an InputExchangeItem and an
OutputExchangeItem (single point, single timestep)
The system driver
• Controls the application flow using a web service
architecture
• Implemented using OpenMI’s Configuration Editor
• Convenient tool for the testing of the OpenMI
implementations and model interactions
Hardware Architecture
• The atmospheric model runs on a high
performance platform, often split into
Compute Notes and Login Nodes
• Access to the Compute Nodes must be
through the Login Nodes
Personal Computer
(Windows)
• Access to the Login Nodes is through
the Virtual Server (Web Svcs)
High Performance Computer
• The Client contains the
OpenMI and SWAT
software, which run on a
Windows platform.
Virtual
Linux
Server
Login Nodes
(kraken)
Compute
Nodes (kraken)
Software Achitecture
Client
• Configuration Editor is
the driver… it is used to
link the models and
trigger the start of the
run.
• Hydrological model
(SWAT 2005) is a
version wrapped to work
with OpenMI
• Access to the
atmospheric model
(CAM) is done through
“wrapper” code that
accesses ESMF web
services via an OpenMI
interface
Personal Computer (Windows)
OpenMI
Configuration Editor
OpenMI
CAM OpenMI
Wrapper
… to Web Services
SWAT 2005
Software Architecture
Server
HPC Login Nodes
Tomcat/Axis2
SOAP Svcs
Process
Controller
Registrar
Linux Server (Web Svr)
• In some high performance
systems, access to nodes can be
restrictive.
• Access to/from external systems
can be controlled via “gateway”
systems using web services.
• Running applications (such as
CAM Component Svc) on
Compute Nodes must be handled
by a job scheduler.
Job
Scheduler
Comp
Svc
Comp
Svc
Comp
Svc
CAM
CAM
CAM
HPC Compute Nodes
Logical Flow - Startup
Personal Computer (Windows)
1
When loading models into the Configuration Editor, each
model is initialized. For CAM, this involves starting a “New
Client” in the Process Controller, which submits a new
CAM Component Service using the Job Scheduler.
1
Config Editor
HPC Login Nodes
3
OpenMI
Process
Controller
SWAT 2005
CAM
Wrapper
5
Registrar
4
Job
Scheduler
2
7
Web Svcs
Linux Server
1.
2.
3.
4.
Initialize
New Client
New Client
Submit Job
5.
6.
7.
Status = SUBMITTED
Instantiate Job (Comp Svc)
Status = READY
6
Comp
Svc
CAM
HPC
Compute Nodes
Logical Flow - Status
The status of the CAM Component Service is checked
often throughout the workflow. The status is stored in the
Registrar, so it can be retrieved via the Process Controller.
Personal Computer (Windows)
Config Editor
HPC Login Nodes
2
OpenMI
CAM
Wrapper
Process
Controller
SWAT 2005
3
Registrar
Job
Scheduler
1
Web Svcs
Linux Server
1.
2.
3.
Get Status
Get Status
Get State
Comp
Svc
CAM
HPC
Compute Nodes
Logical Flow - Initialize
Personal Computer (Windows)
1
Before the models can be run, they need to be initialized.
For CAM, the Initialize call is sent to the CAM Component
Service via Web Services and the Process Controller. The
CAM Component Svc updates it’s status in the Registrar
prior to and after Initialization.
1
Config Editor
HPC Login Nodes
3
OpenMI
Process
Controller
Registrar
SWAT 2005
CAM
Wrapper
Job
Scheduler
2
5
4
6
Web Svcs
Linux Server
1.
2.
3.
4.
Prepare
Initialize
Initialize
Initialize
5.
6.
Status = INITIALIZING
Status = INIT_DONE
Comp
Svc
CAM
HPC
Compute Nodes
Logical Flow – Timestep (Run)
Personal Computer (Windows)
For each Timestep in SWAT, the trigger to run a timestep in
CAM is a Get Values request in the OpenMI Interface. The
Run Timestep request is passed to the CAM Component
Service and the Component Service sets the output data
making it available for later retrieval (see Get Data).
1
Config Editor
HPC Login Nodes
4
OpenMI
2
Process
Controller
Registrar
SWAT 2005
CAM
Wrapper
Job
Scheduler
3
6
5
8
Web Svcs
Linux Server
1.
2.
3.
4.
Get Values
Get Values
Run Timestep
Run Timestep
5.
6.
7.
8.
Run Timestep
Status = RUNNING
Set Output Data
Status = TIMESTEP_DONE
Comp
Svc
CAM
HPC
Compute Nodes
7
Logical Flow – Timestep (Get Data)
Personal Computer (Windows)
After each timestep run, the output data is then fetched
from the CAM Component Service via the Web Services
and Process Controller. The first time fetching data, a
description of the data structure is requested. This
description is then used for the remaining timesteps.
Config Editor
HPC Login Nodes
2
OpenMI
SWAT 2005
CAM
Wrapper
5
Process
Controller
Registrar
3
1
Job
Scheduler
4
6
Web Svcs
Linux Server
1.
2.
3.
4.
Get Data Desc*
Get Data Desc*
Get Data Desc*
Get Data
5.
6.
Get Data
Get Data
* one time only
Comp
Svc
CAM
HPC
Compute Nodes
Logical Flow - Finalize
Personal Computer (Windows)
1
After all timesteps have completed, the models need to be
finalized. For CAM, the Finalize call is sent to the CAM
Component Service via Web Services and the Process
Controller. The CAM Component Svc updates its status in
the Registrar prior to and after finalization.
1
Config Editor
HPC Login Nodes
3
OpenMI
Process
Controller
Registrar
SWAT 2005
CAM
Wrapper
Job
Scheduler
2
5
4
6
Web Svcs
Linux Server
1.
2.
3.
4.
Finish
Finalize
Finalize
Finalize
5.
6.
Status = FINALIZING
Status = FINAL_DONE
End Client (Next Slide)
Comp
Svc
CAM
HPC
Compute Nodes
Logical Flow – End Client
Personal Computer (Windows)
After the Finalize call, the CAM Component Service is
done, so the CAM Wrapper closes it out by calling End
Client. This call results in the CAM Component Service
completing it’s loop and exiting as well as the Process
Controller removing all references to the client.
Config Editor
HPC Login Nodes
2
OpenMI
Process
Controller
5
Registrar
SWAT 2005
CAM
Wrapper
Job
Scheduler
1
3
Web Svcs
Linux Server
1.
2.
3.
4.
End Client
End Client
Kill Server
Exit Service Loop
5.
Status = COMPLETED
Comp
Svc
CAM
HPC
Compute Nodes
4
Logical Workflow
Driver
SWAT/OpenMI
ATM/OpenMI Wrapper
ESMF Web Services
ESMF Component
Initialize
Initialize
NewClient
Prepare
Prepare
GetValues
ValueSet
GetValues
ValueSet
Initialize
ESMF_GridCompInitialize
RunTimestep
ESMF_GridCompRun
GetData
Finish
Finish
Finalize
Dispose
Dispose
EndClient
ESMF_GridCompFinalize
Data Flow
One-Way Coupling
High Performance
Computer
ESMF
Component/CAM
ESMF State
Personal Computer
GetValues
GetDataValues
CAM/OpenMI
Wrapper
SWAT/OpenMI
Input
Exchange Item
Output
Exchange Item
• The data is pulled from the CAM Component to SWAT via the wrapper , initiated by the OpenMI
GetValues call; this call is made once per timestep.
• Data is exchanged between CAM and SWAT using the OpenMI Exchange Item structures that
handle the translation from grid to point values
Data Flow
Two-Way Coupling
High Performance
Computer
ESMF
Personal Computer
GetValues
GetDataValues
Component/CAM
ESMF
Export
State
ESMF
Import
State
SWAT/OpenMI
CAM/OpenMI
Wrapper
Input
Exchange Item
Output
Exchange Item
Input
Exchange Item
SetInputData
• In two-way coupling, each model pulls the data from
the other model using the OpenMI GetValues method.
Extrapolation is used on the first timestep to break the
deadlock between the two model requests.
• OpenMI Input and Output Exchanges items are again
used to exchange and translate the data.
Output
Exchange Item
GetValues
Model Configurations
•
CAM
–
–
–
–
–
–
Global Atmospheric Model
Model run: 1 day
Timestep: 1800 sec
Dynamical Core: finite volume
Horizontal Grid: 10x15
Export data variables:
• surface air temperature
• precipitation
• wind speed
• relative humidity
• solar radiation
•
SWAT
– Hydrology science information
provided by Jon Goodall of
University of S. Carolina
– Lake Fork Watershed (TX)
– Watershed Area: 486.830 km2
– Model run: 2 years, 1977 – 1978
– Timestep = 1 day
– Weather Stations:
• wea62 (33.03 N, 95.92 W)
• wea43 (33.25 N, 95.78 W)
Future Tasks
• Continue development of two-way coupling
• Abstract data exchange within the ESMF wrapper code to
accommodate configuration of different variables for
different model implementations
• Explore coupling via web services provided by other
frameworks and analysis and visualization tools
• Extend SWAT configurations for larger domains, and
examine coupling to other hydrological and impacts models