A Community
Terrain-following
Ocean
Modeling
System
2003 Terrain-Following Ocean Models Users Workshop
PMEL, Seattle, WA, August 5, 2003
Developers and Collaborators
Hernan G. Arango
Rutgers University
Alexander F. Shchepetkin
UCLA
W. Paul Budgell
IMR, Norway
Bruce D. Cornuelle
SIO
Emanuele DiLorenzo
SIO
Tal Ezer
Princeton University
Mark Hadfield
NIWA, New Zealand
Kate Hedstrom
University of Alaska, ARSC
Robert Hetland
TAMU
John Klinck
Old Dominion
Arthur J. Miller
SIO
Andrew M. Moore
University of Colorado
Christopher Sherwood
USGS/WHOI
Rich Signell
SACLANT
John C. Warner
USGS/WHOI
John Wilkin
Rutgers University
Executive Committee
Dale B. Haidvogel
Rutgers University
James C. McWilliams
UCLA
Robert Street
Stanford University
ONR Support
Manuel Fiadeiro
Terri Paluszkiewicz
Charles Linwood Vincent
Objectives
• To design, develop and test an expert ocean modeling system
for scientific and operational applications over a wide range
of scales from coastal to global
• To provide a platform for coupling with operational
atmospheric models, sediment models, and ecosystem models
• To support multiple levels of nesting and composed grids
• To provide tangent linear and adjoint models for variational
data assimilation, ensemble forecasting, and stability analysis
• To provide a framework for massive parallel computations
Approach
• Use state-of-the-art advances in numerical techniques,
subgrid-scale parameterizations, data assimilation,
nesting, computational performance and parallelization
• Modular design with ROMS as a prototype
• Test and evaluate the computational kernel and various
algorithms and parameterizations
• Build a suite of test cases and application databases
• Provide a web-based support to the user community and a
linkage to primary developers
Accomplishments
ROMS/TOMS 2.0 released to beta testers on
January 16, 2003 and full user community on
June 30, 2003.
Built tangent linear and adjoint models and tested
on realistic applications of the US West and East
Coasts: eigenmodes and adjoint eigenmodes,
singular vectors, pseudospectra, forcing singular
vectors, stochastic optimals, and ensemble
forecasting.
The model is used in
oceanographic studies in
over 30 countries by:
• Universities
• Government Agencies
• Research Organizations
(Relief Image from NOAA Animation by Rutgers)
Ocean Modeling
Web Site
http://www.ocean-modeling.org/
Kernel Attributes
• Free-surface, hydrostatic, primitive equation
model
• Generalized, terrain-following vertical coordinates
• Boundary-fitted, orthogonal curvilinear, horizontal
coordinates on an Arakawa C-grid
• Non-homogeneous predictor/corrector timestepping algorithm
• Accurate discretization of the baroclinic pressure
gradient term
• High-order advection schemes
• Continuous, monotonic reconstruction of vertical
gradients to maintain high-order accuracy
Vertical Terrain-following Coordinates
Vieste
(Italy)
Dubrovnik
(Croatia)
Depth
(m)
Longitude
Curvilinear Transformation
Cartesian
Spherical
Polar
Code Design
• Modular, efficient, and portable F90/F95 Fortran
code with dynamical allocation of memory via dereferenced pointer structures.
• C-preprocessing managing
• Multiple levels of nesting and composed grids
• Lateral boundary conditions options for closed,
periodic, and radiation
• Arbitrary number of tracers (active and passive)
• Input and output NetCDF data structure
• Support for parallel execution on both shared- and
distributed -memory architectures
Model Grid Configuration
Nested
Composed
Parallel Framework
• Coarse-grained parallelization
Parallel Tile Partitions
8x8
Ny
} Nx
Parallel Framework
• Coarse-grained parallelization
• Shared-memory, compiler depend directives
MAIN (OpenMP 2.0 standard)
• Distributed-memory (MPI)
• Optimized for cache-bound computers
• ZIG-ZAG cycling sequence of tile partitions
• Few synchronization points
• Serial and Parallel I/O (via NetCDF)
• Efficiency 4-64 threads
(Ezer)
The cost of saving output
and global averaging is
much higher for the MPI
code
(for the shared-memory SGI
machine)
(Ezer)
Subgrid-Scale Parameterizations
• Horizontal mixing of tracers along level, geopotential,
isopycnic surfaces
• Transverse, isotropic stress tensor for momentum
• Local, Mellor-Yamada, level 2.5, closure scheme
• Non-local, K-profile, surface and bottom closure
scheme
• Local, Mellor-Yamada, level 2.5, closure scheme
• General Length-Scale turbulence closure (GOTM)
Boundary Layers
• Air-Sea interaction boundary layer from COARE
(Fairall et al., 1996)
• Oceanic surface boundary layer (KPP; Large et al.,
1994)
• Oceanic bottom boundary layer (inverted KPP;
Durski et al., 2001)
• Wave / Current / Sediment bed boundary layer
(Styles and Glenn, 2000; Blaas; Sherwood)
Modules
• Lagrangian Drifters (Klinck, Hadfield, Capet)
• Tidal Forcing (Hetland, Signell)
•
River Runoff (Hetland, Signell, Geyer)
• Sea-Ice (Budgell, Hedstrom)
• Biology Fasham-type Model (Moisan, Di Lorenzo,
Shchepetkin, Frenzel, Fennel, Wilkin)
• EcoSim Bio-Optical Model (Bissett, Wilkin)
• Sediment erosion, transport and deposition (Warner,
Sherwood, Blaas)
Ongoing and Future Work
• One- and two-way nesting
• Wetting and drying capabilities
• Sediment model
• Bottom boundary layer models
• Ice model
• Parallelization of adjoint model
• Variational data assimilation
• Parallel IO
• Framework (ESMF)
• Web-based dynamic documentation
• Test cases
• WRF coupling
One-Way Nesting
North Atlantic Basin
•
•
•
•
•
1/10 degree resolution (1002x1026x30)
Levitus Climatology
NCEP daily winds: 1994-2000
COADS monthly heat fluxes
Requirements:
• Memory: 11 Gb
• Input data disk space: 16 Gb
• Ouput data disk space: 280 Gb
• 32 Processors Origin 3800, 4x16
• CPU: 46 hours per day of simulation
• Wall clock: 153 days for 7-year simulation
Bathymetry
(m)
r-factor = 3.2
1/10 degree
Resolution
ETOPO5
Free-Surface
(m)
Temperature
at 100 m
US East Coast
•
•
•
•
30 km resolution (192x64x30)
Initialized for North Atlantic Basin Simulation
NCEP daily winds
COADS monthly heat fluxes with imposed daily
shortwave radiation cycle.
• One-way nesting
• Boundary conditions from 3-day averages
• Flather/Chapman OBC for 2D momentum
• Clamped OBC for 3D momentum and tracers
• Rivers
• Fasham-type biology model
Temperature
at 100 m
One-Way
Coupling
(Wilkin)
Potential
Temperature
at 50 m
(Celsius)
30 km
Resolution
(Wilkin)
Surface
Temperature
Surface
Chlorophyll
Publications
Ezer, T., H.G. Arango and A.F. Shchepetkin, 2002: Developments in Terrain-Following Ocean Models: Intercomparisons of Numerical
Aspects, Ocean Modelling, 4, 249-267.
Haidvogel, D.B., H.G. Arango, K. Hedstrom, A. Beckmann, P. Malanotte-Rizzoli, and A.F. Shchepetkin, 2000: Model Evaluation
Experiments in the North Atlantic: Simulations in Nonlinear Terrain-Following Coordinates, Dyn. Atmos. Oceans, 32, 239-281.
MacCready, P. and W.R. Geyer, 2001: Estuarine Salt Flux through an Isoline Surface, J. Geoph. Res., 106, 11629-11639.
Malanotte-Rizzoli, P., K. Hedstrom, H.G. Arango, and D.B. Haidvogel, 2000: Water Mass Pathways Between the Subtropical and Tropical
Ocean in a Climatological Simulation of the North Atlantic Ocean Circulation, Dyn. Atmos. Oceans, 32, 331-371.
Marchesiello, P., J.C. McWilliams and A.F. Shchepetkin, 2003: Equilibrium Structure and Dynamics of the California Current System, J.
Phys. Oceanogr., 34, 1-37.
Marchesiello, P., J.C. McWilliams, and A.F. Shchepetkin, 2001: Open Boundary Conditions for Long-Term Integration of Regional Ocean
Models, Ocean Modelling, 3, 1-20.
Moore, A.M., H.G. Arango, A.J. Miller, B.D. Cornuelle, E. Di Lorenzo, and D.J. Neilson, 2003: A Comprehensive Ocean Prediction and
Analysis System Based on the Tangent Linear and Adjoint Components of a Regional Ocean Model, Ocean Modelling, Submitted.
Peven, P., C. Roy, A. Colin de Verdiere and J. Largier, 2000: Simulation and Quantification of a Coastal Jet Retention Process Using a
Barotropic Model, Oceanol. Acta, 23, 615-634.
Peven, P., J.R.E. Lutjeharms, P. Marchesiello, C. Roy and S.J. Weeks, 2001: Generation of Cyclonic Eddies by the Agulhas Current in the
Lee of the Agulhas Bank, Geophys. Res. Let., 27, 1055-1058.
Shchepetkin, A.F. and J.C. McWilliams, 2003: The Regional Ocean Modeling System: A Split-Explicit, Free-Surface Topography-Following
Coordinates Ocean Model, J. Comp. Phys., Submitted.
Shchepetkin, A.F. and J.C. McWilliams, 2003: A Method for Computing Horizontal Pressure-Gradient Force in an Oceanic Model with a
Non-Aligned Vertical Coordinate, J. Geophys. Res., 108, 1-34.
She, J. and J.M. Klinck, 2000: Flow Near Submarine Canyons Driven by Constant Winds, J. Geophys. Res., 105, 28671-28694.
Warner, J.C., H.G. Arango, C. Sherwood, B. Butman, and Richard P. Signell, 2003: Performance of four turbulence closure methods
Implemented using a Generic Length Scale Method, Ocean Modelling, Revised and Resubmitted.
Modular Design
Code Design
#include "cppdefs.h“
MODULE mod_ocean
USE mod_kinds
implicit none
TYPE T_OCEAN
real(r8),
real(r8),
real(r8),
real(r8),
real(r8),
real(r8),
#ifdef SOLVE3D
real(r8),
real(r8),
real(r8),
real(r8),
real(r8),
real(r8),
real(r8),
real(r8),
real(r8),
# ifdef SEDIMENT
real(r8),
real(r8),
real(r8),
# endif
#endif
pointer
pointer
pointer
pointer
pointer
pointer
::
::
::
::
::
::
rubar(:,:,:)
rvbar(:,:,:)
rzeta(:,:,:)
ubar(:,:,:)
vbar(:,:,:)
zeta(:,:,:)
pointer
pointer
pointer
pointer
pointer
pointer
pointer
pointer
pointer
::
::
::
::
::
::
::
::
::
pden(:,:,:)
rho(:,:,:)
ru(:,:,:,:)
rv(:,:,:,:)
t(:,:,:,:,:)
u(:,:,:,:)
v(:,:,:,:)
W(:,:,:)
wvel(:,:,:)
pointer :: bed(:,:,:,:)
pointer :: bed_frac(:,:,:,:)
pointer :: bottom(:,:,:)
END TYPE T_OCEAN
TYPE (T_OCEAN), allocatable :: ALL_OCEAN(:)
END MODULE mod_ocean
CONTAINS
SUBROUTINE allocate_ocean (ng, LBi, UBi, LBj, UBj)
USE mod_param
#ifdef SEDIMENT
USE mod_sediment
#endif
integer, intent(in) :: ng, LBi, UBi, LBj, UBj
IF (ng.eq.1) allocate ( OCEAN(Ngrids) )
allocate
allocate
allocate
allocate
allocate
allocate
(
(
(
(
(
(
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
%
%
%
%
%
%
rubar(LBi:UBi,LBj:UBj,2) )
rvbar(LBi:UBi,LBj:UBj,2) )
rzeta(LBi:UBi,LBj:UBj,2) )
ubar(LBi:UBi,LBj:UBj,3) )
vbar(LBi:UBi,LBj:UBj,3) )
zeta(LBi:UBi,LBj:UBj,3) )
#ifdef SOLVE3D
allocate
allocate
allocate
allocate
allocate
allocate
allocate
allocate
(
(
(
(
(
(
(
(
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
OCEAN(ng)
%
%
%
%
%
%
%
%
pden(LBi:UBi,LBj:UBj,N(ng)) )
rho(LBi:UBi,LBj:UBj,N(ng)) )
ru(LBi:UBi,LBj:UBj,0:N(ng),2) )
rv(LBi:UBi,LBj:UBj,0:N(ng),2) )
t(LBi:UBi,LBj:UBj,N(ng),3,NT(ng)) )
u(LBi:UBi,LBj:UBj,N(ng),2) )
v(LBi:UBi,LBj:UBj,N(ng),2) )
W(LBi:UBi,LBj:UBj,0:N(ng)) )
# ifdef SEDIMENT
allocate ( OCEAN(ng) % bed(LBi:UBi,LBj:UBj,Nbed,MBEDP) )
allocate ( OCEAN(ng) % bed_frac(LBi:UBi,LBj:UBj,Nbed,NST) )
allocate ( OCEAN(ng) % bottom(LBi:UBi,LBj:UBj,MBOTP) )
# endif
#endif
RETURN
END SUBROUTINE allocate_ocean
SUBROUTINE initialize_ocean (ng, tile)
USE mod_param
#ifdef SEDIMENT
USE mod_sediment
#endif
integer, intent(in) :: ng, tile
integer :: IstrR, IendR, JstrR, JendR, IstrU, JstrV
real(r8), parameter :: IniVal = 0.0_r8
#include "tile.h"
#ifdef DISTRIBUTE
IstrR=LBi
IendR=UBi
JstrR=LBj
JendR=UBj
#else
# include "set_bounds.h"
#endif
OCEAN(ng) % rubar(IstrR:IendR,JstrR:JendR,1:2) = IniVal
OCEAN(ng) % rvbar(IstrR:IendR,JstrR:JendR,1:2) = IniVal
OCEAN(ng) % rzeta(IstrR:IendR,JstrR:JendR,1:2) = IniVal
OCEAN(ng) % ubar(IstrR:IendR,JstrR:JendR,1:3) = IniVal
OCEAN(ng) % vbar(IstrR:IendR,JstrR:JendR,1:3) = IniVal
OCEAN(ng) % zeta(IstrR:IendR,JstrR:JendR,1:3) = IniVal
...
RETURN
END SUBROUTINE initialize_ocean
END MODULE mod_ocean
#include "cppdefs.h"
MODULE omega_mod
implicit none
PRIVATE
PUBLIC omega
CONTAINS
SUBROUTINE omega (ng, tile)
USE mod_param
USE mod_grid
USE mod_ocean
integer, intent(in) :: ng, tile
# include "tile.h“
# ifdef PROFILE
CALL wclock_on (ng, 13)
# endif
CALL omega_tile (ng, Istr, Iend, Jstr, Jend,
&
LBi, UBi, LBj, UBj,
&
GRID(ng) % Huon,
&
GRID(ng) % Hvom,
&
GRID(ng) % z_w,
&
OCEAN(ng) % W)
# ifdef PROFILE
CALL wclock_off (ng, 13)
# endif
RETURN
END SUBROUTINE omega
&
&
&
&
&
SUBROUTINE omega_tile (ng, Istr, Iend, Jstr, Jend,
&
LBi, UBi, LBj, UBj,
&
Huon, Hvom, z_w, W)
&
&
USE mod_param
USE mod_scalars
USE bc_3d_mod, ONLY : bc_w3d_tile
integer, intent(in) :: ng, Iend, Istr, Jend, Jstr
integer, intent(in) :: LBi, UBi, LBj, UBj
real(r8),
real(r8),
real(r8),
real(r8),
intent(in) :: Huon(LBi:,LBj:,:)
intent(in) :: Hvom(LBi:,LBj:,:)
intent(in) :: z_w(LBi:,LBj:,0:)
intent(out) :: W(LBi:,LBj:,0:)
integer :: IstrR, IendR, JstrR, JendR, IstrU, JstrV
integer :: i, j, k
real(r8), dimension(PRIVATE_1D_SCRATCH_ARRAY) :: wrk
# include "set_bounds.h"
DO j=Jstr,Jend
DO i=Istr,Iend
W(i,j,0)=0.0_r8
END DO
DO k=1,N(ng)
DO i=Istr,Iend
W(i,j,k)=W(i,j,k-1)&
(Huon(i+1,j,k)-Huon(i,j,k)+
&
Hvom(i,j+1,k)-Hvom(i,j,k))
END DO
END DO
...
END DO
RETURN
END SUBROUTINE omega_tile
&
&