DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
A Super-Regional Testbed to Improve Models of Environmental Processes on
the U.S. Atlantic and Gulf of Mexico Coasts
Principal Investigator: L.D. Wright, Director of Coastal and Environmental Research, SURA
wright@sura.org
Award Number:
NA10NOS0120063
The goal of this IOOS-funded project is to create a multi-disciplinary, community-modeling testbed
that will facilitate improved operational coastal ocean prediction. The testbed will allow scientists to
share models, observations, and tools needed to elucidate, prioritize, and resolve issues associated with
interoperable coupling of models. The goals include improving and evaluating models already in
operational use within the Atlantic and Gulf Coast super-region, developing and assessing metrics for
model skill and system performance, and facilitating the transition of models from research to
operations. Pilot testbed projects will address three chronic issues of high relevance within the super
region: (1) coastal inundation (2) estuarine hypoxia and (3) shelf hypoxia. An overarching theme will
involve the design and implementation of cyberinfrastructure (CI) to support the three issue-motivated
science themes. From the outset, an unstated goal was for this project to become a showcase for
federal-academia-industry collaboration. As the team has come together, it is apparent that we are well
on track to realizing that goal. The testbed involves 20 universities, 11 federal centers and programs and
2 private corporations.
This report will highlight the accomplishments, progress and challenges of each of the testbed teams
and the SURA management activity for the period of August 2010 and is organized as follows:
1. SURA Management
2. Cyberinfrastructure Task
3. Shelf Hypoxia Modeling in the Gulf of Mexico
4. Estuarine Hypoxia Modeling in Chesapeake Bay
5. Coastal Inundation Modeling in the Gulf of Mexico and the Gulf of Maine
Appendix A: Testbed High Level Deliverables
Appendix B: TAEG Comments on Testbed Leadership telecon on 31 August 2010
Compiled by Elizabeth Smith (lsmith@sura.org)
1
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
1. SURA Management
SURA is providing overall project leadership, multi-institution coordination and management of the
testbed and is insuring that federal funds are expended in a responsible and accountable manner. SURA
has issued subcontract agreements to our test bed partners, and has communicated the expectation
that invoices are to be submitted on a monthly basis. SURA has been monitoring, approving and paying
the invoices as they are submitted through SURA’s financial and accounting services.
A dedicated SURA Linux server has been set up and is rapidly being populated with examples of model
output for the CI team to assess, and to use in their development of a catalog of models. A
collaborative web site has been created by CI team-member, University of Alabama Huntsville, to
facilitate communication and information exchange: http://testbed.sura.org. SURA continues to work
closely with the website administrators and developers to provide feedback on the usability of the site.
SURA has been facilitating and participating in all modeling team and CI teleconferences, which have
been occurring regularly, and has recently made available a WebEx for on-line meetings. SURA, with the
team leads and Testbed Advisory and Evaluation Group (TAEG) has developed a suite of high-level
testbed deliverables, or milestones, intended for a broad audience to highlight, in non-technical terms,
what will be the major outcomes of the testbed. These deliverables are found in an Appendix A.
SURA is convening monthly teleconferences among the modeling team and CI leaders (Friedrichs,
Harding, Howlett, Luettich), and the Testbed Advisory and Evaluation Group (TAEG) to discuss progress
and challenges, and to give the TAEG an opportunity to ask questions and provide feedback and
guidance. Subsequently, the TAEG held a teleconference during which they indicated their general
satisfaction with the initial progress of the testbed. Appendix B contains the notes from the TAEG
comments on the Testbed Leadership conference help on 31 August 2010.
SURA IT Progress Report Input
The primary focus area for SURA IT staff this past month was increasing the pool of high performance
computational resources available to the project teams. The model teams indicated that their current
resources were insufficient to handle large scale data sets and complete model runs in a timely manner.
In response, SURA polled the testbed groups regarding their need for additional computational
resources. The inundation modeling group responded with specific requirements. Members of the
Inundation Team expressed the need for access to computational capacity beyond that available at their
local institutions included; Harry Wang (VIMS), Chunyan Li (LSU), Lianyuan Zheng (USF) and William
Perrie (Bedford Institute of Oceanography).
To support the needs of the inundation team SURA obtained startup allocations on a large shared HPC
system located at LSU (Queen Bee) by working directly with staff at LSU and the Louisiana Optical
Network Initiative (LONI). In addition a TeraGrid startup allocation was obtained on the Ranger cluster at
Texas Advanced Computing Center (TACC). Ranger is one of the largest computing systems in the world
for open science research. To date all four researchers from the inundation team have gained access to
and are using the resources at LONI, including Queen Bee. We are in the process of establishing TeraGrid
user accounts for the same four inundation team members so they will be able to transition to the larger
allocations on TeraGrid when necessary. After only a couple of weeks of access the inundation team has
consumed about half of the initial LONI startup allocation of 50,000 SUs. In anticipation of the
inundation group’s continued need for HPC resources, SURA submitted a proposal for a large allocation
Compiled by Elizabeth Smith (lsmith@sura.org)
2
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
of 200,000 SUs on the LONI Queen Bee system. This request is now in review by the LONI allocations
committee, and if accepted, this allocation will be available starting October 1st. The request was
coordinated with Rick Luettich and Liz Smith to ensure the description of the science, models and
methodology descriptions were accurate.
SURA IT staff members are also working with the project CI Team and participants from the University of
Alabama Huntsville to help ensure that the model data and tools that are being stored on project
testbed servers will be accessible through the Testbed collaboration website.
Compiled by Elizabeth Smith (lsmith@sura.org)
3
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
2. Cyberinfrastructure (CI) Testbed – Progress as of 9/7/2010
(Submitted by Eoin Howlett on behalf of the CI Team)
General
1. Work has been focused on identifying the data that will be coming from the model teams and
aggregating the model results on the Sura server. The Sura Linux server was made available and we
have been populating the server with data and software.
2. Some model data was still being sent to the older server, we confirmed that data needs to be
uploaded to the Linux server (http://testbed.sura.org/node/100)
host: testbedapps.sura.org
user: sura_ftp
pass: 3ur4_7tp
port: 22 (sftp)
Estuarine Hypoxia
1. Completed translation of the 20 year Ch3d hindcast for the Chesapeake Bay in binary format
into a CF compliant NetCDF standard. The data was sent to Hortensia at LSU and has been
posted on the Sura server and is available via TDS and ncWMS.
Ch3d NetCDF data published with TDS/ncWMS
Compiled by Elizabeth Smith (lsmith@sura.org)
4
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
2. Status of model data on testbed server
Model
Source
Estuarine
Hypoxia
Sample on SURA
server
CBOFS2
NOAA
NetCDF
ChesROMS
UMCES
NetCDF
CH3D
USACE
NetCDF
EFDC
VIMS
ASCII
Information on sample output
Notes
http://testbedapps.sura.org/thredds/fileS
erver/efdc/Readme.txt
Dave Forrest
working on
converter, will be
similar to elcirc.
Information on sample output
Notes
Inundation
1. Status of model data on testbed server
Model
Source
Sample on SURA
server
Inundation
WW3 (3
grids)
BIO
WW3 Format
SLOSH
UF
None
FVCOM Suituate
ELCIRC
SELFE
UMASS
VIMS
VIMS
NetCDF
http://testbedapps.sura.org/thredds/fileS
erver/bio_ww3/ww3_extratropical/READ
ME_Noel_Hindcast.doc
WW3 can
produce Grib –
can they turn it
on? Howard to
check.
SWAN also?
Don Slinn is
contact, Eoin will
contact.
http://testbedapps.sura.org/thredds/fileS
erver/umass_fvcom/fvcom_extratropical/
read_me_file.txt
None
Dave Forrest working on adapting the
ELCIRC => NetCDF translator for use with
SELFE
http://testbedapps.sura.org/thredds/fileS
erver/und_adcirc/adcirc_tropical/adcirc_r
eadme.txt
Grid for SURA will
be pure triangles.
Sample is mixed
tri/quad.
Check with
Battista’s group,
they have tools.
Howard to
contact UND.
Check if running
coupled SWAN
and check that
NetCDF can be
created
Listing of provided output files here:
Justin to contact,
NetCDF
ADCIRC
UND
ADCIRC Format v49
FVCOM
USF
Unknown Format
Compiled by Elizabeth Smith (lsmith@sura.org)
5
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
http://testbedapps.sura.org/thredds/catal
og/usf_fvcom/fvcom_tropical/catalog.ht
ml
we need NetCDF.
2. Resource Issues. The inundation team is planning to run models on several computational grids,
including a Gulf of Mexico grid, which is order millions of nodes. Running the individual models on a
grid this complex requires more powerful cluster resources than many of the team members have
readily available. The inundation team members, with the assistance of SURA staff, are in the
process of identifying and provisioning additional computational resources. In addition there has
been some discussion of simplifying the computational grids used. The CI team is monitoring these
efforts and will attempt to lend expertise and experience as possible and needed.
3. We have completed the initial stage of the model output file format assessment for the inundation
models. As seen from the table above, many of the inundation team models do not support NetCDF
format natively. Based on this analyses, the CI team will develop a set of tasks for converting model
output to a standard format for comparison.
Shelf Hypoxia
1. Status of model data on testbed server
Model
Source
Sample on SURA
server
Information on sample output
NetCDF
http://edacdap.northerngulfinstitute.org/thredds/ca
talog/ncom/amseas_download/catalog.h
tml
Notes
Shelf Hypoxia
NCOM AMSEAS
NAVOC
EAN
FVCOM
NOAA/
NOS
POM
NOAA/
NOS
HYCOM
NRL
None
NetCDF - One file per
variable
NRL
NetCDF - One file per
variable
NRL
NetCDF
http://tds.hycom.org/thredds/GOMl0.04
/expt_20.1.html
http://edacdap.northerngulfinstitute.org/thredds/ca
talog/iasnfs/catalog.html
http://edacdap.northerngulfinstitute.org/thredds/ca
talog/ncom_gomex/catalog.html
TAMU
NetCDF
http://megara.tamu.edu:8080/thred
ds/ncss/grid/gcoos/hypoxia/dataset.
NCOM IASNFS
NCOM Gulf of
Mexico
ROMS
NOS running
experimentally –
Frank Aikman
Rich Patchen’s
Model – available
on TDS?
None
Compiled by Elizabeth Smith (lsmith@sura.org)
6
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
html
Other
1. Rich Signell to meet at ASA on Sept 8th to discuss unstructured grid implementation
2. Eve Devaliere (from Jeff Hanson’s office) will be at ASA Sept 7-10 to work with ASA staff on
implementation of skill assessment toolbox with NetCDF/TDS access. The initial plan for the
Matlab-based IMEDS toolbox is:
a. Select observation data the same way as IMEDS currently does
b. Select the model we want to validate – this would be linked to the url of the
corresponding opendap server – we will start with CH3D water-level
c. Based on the observation data, IMEDS would identify the the I,J grid cells from the
model grid and give the user the option to pick the desired time span within the model’s
time
d. Extract and download the time-series from the grid and convert it into wavedat file
e. Process as usual and produce metrics
3. Web Site/Catalog sub-group (Eoin, Charlton, Eric, Kyle) had con-call on web site and catalog,
notes below. Focus on re-use of Ted Haberman (NOAA) work to manage TDS catalogs and
metadata and portal work using GeoEXT. Preliminary portal is available at
http://testbedwww.sura.org/model/ - notes attached as appendix.
Compiled by Elizabeth Smith (lsmith@sura.org)
7
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Sample Model Testbed site using SURA testbed catalog
4. TDS software configured and operational on SURA Server and is live at
http://testbedapps.sura.org/thredds/catalog.html
The following data sets are TDS compatible:
http://testbedapps.sura.org/thredds/catalog/samples/catalog.html
ERDDAP has been configured and operational on SURA Server and is live at
http://testbedapps.sura.org/erddap/
Next Month:
5. Complete model catalog with closure on what models will be included. For models that do not
have output in “IOOS standards”, develop plan to implement that.
6. Skill assessment meeting (Sept 7-10) and implementation
7. Work on observation catalog for comparisons/skill assessment
8. Work on web site/live catalog
9. Unstructured Grid work plan (meeting Sept 8)
10. Evaluate Ramadda from Unidata
11. Work on tools to make non-compliant model data compliant
Challenges:
1. Lack of clarity on:
a. Whether all models will be on SURA server, some distributed?
b. Will all model data be in gridded NetCDF, some discussion that some models may just
produce time series output at certain points
c. Where the observation data will come from for comparison and assessment
2. Finalize how tools and model preview web site will be connected to Drupal collaboration site.
Compiled by Elizabeth Smith (lsmith@sura.org)
8
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
3. Shelf Hypoxia Modeling – Progress as of 8/26/2010
(Submitted by John Harding on behalf of the Shelf Hypoxia Team)
Overall Project Management – John Harding (Lead)
Tasks from work plan:
- Oversee sub-project task progress & milestone completion
- Provide monthly status report for SURA management
Progress/ Accomplishments (August 2010):
August Shelf Hypoxia Team telcon conducted with review of sub-projects and identification of
necessary action items by sub-project team leads
Planned Work (September 2010) based on work plan
Work with sub-project team leads to monitor September action items
Conduct September Shelf Hypoxia Team telcon
Issues/ Challenges
See Hetland post-doc issue identified under sub-project 1 & 2
See Wiggert student issue identified under sub-project 2
Sub-project 1 – Assess hydrodynamic skill – Rob Hetland (Lead)
Tasks from work plan:
- Identify / assemble physical evaluation data (Due 31 Aug)
- Compile/organize/store data (Due monthly beginning July 31, ending February 28)
- Provide NGOM POM hindcast output (Due 31 July) – Delayed to September
- Provide IASNFS NCOM hindcast output (Due 31 July) – Delayed to September
- Provide HYCOM GOM hindcast output (Due 31 July) – Task Complete
- Couple Shelf ROMS to NGOM POM (Due 30 Sep) – See Challenges/Issues
- Couple Shelf ROMS to NGOM POM (Due 30 Sep) – See Challenges/Issues
- Couple Shelf ROMS to IASNFS (Due 30 Sep) – See Challenges/Issues
- Couple Shelf ROMS to GOM HYCOM (Due 30 Sep) – See Challenges/Issues
- Run uncoupled Shelf ROMS (Due 30 Sep) – See Challenges/Issues
- Evaluate/ develop physical skill assessment tools (Due August 31, November 30, December 31)
Progress/ Accomplishments (August 2010):
Identify/compile/organize/store of physical evaluation data continues.
Harding & Ko met to discuss 2004-2009 IASNFS transfer as soon as new EDAC / OceanNOMADS
server operational.
New EDAC/ OceanNOMADS servers delivered to MSU week of August 23.
Compiled by Elizabeth Smith (lsmith@sura.org)
9
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Hetland postdoc identified and hired to conduct coupling and hindcast runs. Expected start date
mid-October
Planned Work (September 2010) based on work plan
Matt Howard will contact John Lehrter to gain access to EPA data
Matt Howard will provide Bruce Lipphardt pointers to model output and data access
Rob Hetland will provide sample shelf hypoxia output to Matt Howard for copy to SURA
THREDDS server
Eugene Wei contacting Rich Patchen for September delivery of 2004-2009 NGOM POM to Matt
Howard for to conversion NetCDF/OPeNDAP and storage on SURA server.
Harding/ Ko will undertake transfer of 2004-2009 IASNFS from NRL to OceanNOMADS server.
Rob Hetland & Katja Fennel to confer and determine work plan modifications based on Hetland
post-doc issue identified below.
Matt Howard to contact Rich Patchen to arrange for transition of CSDL model evaluation tools to
SURA server.
Issues/ Challenges
Original Hetland post–doc hire fell through. New post-doc identified and hired but won’t start
until mid-October. This will likely delay model coupling and hindcast runs originally scheduled for
completion by end of November and may impact schedules of Subprojects 1 & 2. Task 1 & 2 leads will
confer on work plan modifications to accommodate this delay.
High Priority Items Identified for Year 2
Use of larger domain shelf hypoxia grid for potential year 2 hindcasts
Sub-project 2 – Assess ecosystem model skill – Katja Fennel (Lead)
Tasks from work plan:
- Identify / assemble hypoxia-specific evaluation data (Due 31 Aug, monthly thereafter until February 28)
- Compile/organize/store data (Due monthly beginning July 31, ending February 28)
- Evaluate/ develop biogeochemical skill assessment tools (Due August 31, November 30, December 31)
Progress/ Accomplishments (August 2010):
We performed two simulations for the Texas-Louisiana shelf domain with our biogeochemical
model. The simulations use climatological boundary conditions, and we refer to the first of these as the
baseline. The only difference to the second simulation is in the wind forcing. While the baseline uses
high-frequency winds form a buoy in the Gulf, the second uses NARR winds. Chlorophyll-based metrics
were calculated for both (see below).
We acquired 4-km monthly mean SeaWiFS and MODIS-Aqua chlorophyll products from the
NASA Goddard Server and interpolated these to our model grid. Based on these data we calculated a
time series of metrics: the ratio of standard deviations of model and satellite, bias, root-mean-square
error, and model efficiency (or skill) for the two simulations describe above (see example in Figure 1).
Compiled by Elizabeth Smith (lsmith@sura.org)
10
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
We began compiling other available data, specifically dissolved oxygen (DO), and participated in
a discussion via e-mail and telecon, which involved Russ Beard at NOAA NODC. We will continue to
interact with this group in our efforts to compile relevant data. An overview of the stations with DO
currently available to us is given in Figure 2.
Figure 1: Metrics based in satellite chlorophyll. 050 refers to the baseline simulation. 051 uses NARR
winds instead.
Compiled by Elizabeth Smith (lsmith@sura.org)
11
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Figure 2: Number of stations with dissolved oxygen (DO) measurements available to us thus far.
LUMCON refers to data from Nancy Rabalais’ group. MCH refers to data from Steve DiMarco’s projects.
Planned Work (September 2010) based on work plan
Matt Howard will contact John Lehrter to gain access to EPA data (See also sub-project 1)
Katja Fennel, with team, will focus on determining additional metrics that are more directly
related to hypoxia, including hypoxic area, based on the data currently available and will continue in our
data compilation efforts.
Eugene Wei will contact Rich Patchen about getting the hindcast COAMPS fields to Matt for
translation to THREDDS and inclusion on the SURA server
Katja Fennel and DOng Shan Ko will correspond on what he has available and how to best
conduct their hypoxia model comparison.
Jerry will work with Scott Cross (NCDDC) on test case getting Alan Shiller's DO and other data
onto NODC site.
Katja Fennel will contact Ruoying He about using his biogeochemical model output for our
boundary conditions (biogeochemical variables only).
Compiled by Elizabeth Smith (lsmith@sura.org)
12
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Issues/ Challenges
The delay in hiring a post-doc for Hetland’s group will affect the ecosystem skill component of
this project if it delays the provision of physical boundary conditions.
High Priority Items Identified for Year 2
None at this time.
Sub-project 3 – Evaluate/transition pre-operational AMSEAS NCOM
– Jerry Wiggert (Lead)
Tasks from work plan:
- Identify / assemble real-time ocean & atmospheric evaluation data (Due 31 Aug, monthly thereafter
through May 31)
- Provide real-time AMSEAS NCOM output (Due June 30, monthly thereafter through May 31)
- Store NAVO tools and accumulate AMSEAS on NGI/NCDDC OceanNOMADS (Due July 31, monthly
thereafter through 31 May)
- Evaluate/develop additional real time skill assessment tools (Due 30 November)
- Evaluate AMSEAS (Due 31 March)
- Evaluate AMSEAS COAMPS forcing (Due 31 March)
Progress/ Accomplishments (August 2010):
New EDAC/ OceanNOMADS servers delivered to MSU week of August 23.
Continue in situ data compilations from NAVO (glider, ctd, xbt). In hand are 17-Feb-2010
through 12-Aug-2010
Continue accumulation of pre-operational 3 hrly AMSEAS NCOM 4 day forecasts on NGI/NCDDC
EDAC in OPenDAP format beginning 25 May.
Past OPTEST Reports and preliminary analysis of AMSEAS-NCOM provided by Frank Bub include
a number of metrics examples that are the starting point for tools to apply.
These include:
T and S comparisons at depths (10, 50, 100, 200, 500 and 1000 meters).
∆T and ∆S (i.e., model-obs) at these depths.
T and S scatter plots of model vs. obs .
Acoustic properties
Sonic Layer Depth (SLD)
In-layer sound speed gradient (ILG)
Below-layer sound speed gradient (BLG)
SLD Cut-off Frequency (COF)
Additional metrics under consideration
Taylor diagrams
Target diagrams
Assessment metrics being employed by Estuarine Group
Compiled by Elizabeth Smith (lsmith@sura.org)
13
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Planned Work (September 2010) based on work plan
NGI staff & Harding will set up new EDAC/ OceanNOMADS servers
Accumulation of Assessment Data
Wiggert will coordinate ongoing acquisition of NAVO compilations w/ Frank Bub.
Wiggert will continue accumulation of remote sensing data streams (SST and SSH)
Wiggert will work with Howden (USM) to obtain CODAR and CenGOOS mooring data
Develop Metrics Toolkit for AMSEAS evaluation
Wiggert will continue effort to identify useful metrics for model skill evaluation
Wiggert will connect with Bruce Lipphardt for his input on useful assessment metrics to include
Develop Facility with MAVE/PAVE tools
Wiggert will continue to familiarize with this Matlab-based toolkit and bring programmer (J.
Davis) up to speed. As needed, Frank Bub will be tapped for assistance
Coordinate w/ Scott Cross on test case for moving MAGMIX (Alan Shiller) cruise data on to NODC
repository
Wiggert/Shiller will evaluate how this process works and suggest any ways process may be
simplified. Metadata aspect of transferring data to NODC/NCDDC is a particular issue to assess.
Issues/ Challenges
Initially, we expected that Wiggert's student (B. Denton) would be primary Matlab programmer.
However, she has obtained separate funding for her PhD studies so an alternative was needed. Need
thus developed to identify individual to take on programming tasks. Role of research associate, Jamie
Davis (USM) will be expanded to be primary programmer for USM's effort. He had already been slated
for serving in a more advisor/assistance role. Bringing Jamie in as lead is probably a more appropriate
use for the model assessment activity assigned to USM. It will promote more timely generation of
results and he works closely with Howden on the CODAR array and CenGOOS mooring. Availability of his
time has been confirmed.
High Priority Items Identified for Year 2
Identification and archiving of NAVO, NOAA, and other Deepwater Horizon data for use in any
Year 2 proposal
Compiled by Elizabeth Smith (lsmith@sura.org)
14
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
4. Estuarine Hypoxia – Progress as of 9/3/2010
(Submitted by Carl Friedrichs on behalf of the Estuarine Hypoxia Team)
Estuarine Hypoxia Update, draft version September 3, 2010
Progress in August:
-- In collaboration with the CI team, the full CH3D output for Chesapeake Bay hydrodynamics for 19852005 has been converted to NetCDF.
-- The CBOFS2 synoptic hindcast for 2004-2005 Chesapeake Bay hydrodynamics has been submitted to
the SURA server in NetCDF format, including the model grid, initialization fields and forcings.
-- ChesROMS forcing and model output (3D fields daily and 128 stations hourly) for 1991-2005 has been
submitted to the SURA server in NetCDF format.
-- EFDC model simulations for Chesapeake Bay hydrodynamics for 2004 have been submitted to the
SURA server in ASCII format.
-- A model intercomparison of base runs for CH3D, CBOF2, ChesROMS and EFDC for salinity at 28
stations in the Chesapeake Bay for all of 2004 has been completed and presented to the Estuarine
Hypoxia Team in the form of an REU (NSF Research Experience for Undergraduates) report. Systematic
model-data mismatch was found in all models as a function of station latitude and/or salinity suggesting
oceanic and/or riverine forcing can be improved.
-- UMCES ROMS has completed hydrodynamic simulations for 2004-2005 (output is NetCDF).
-- A 100K+ unstructured grid of the Chesapeake Bay has been completed for use with SELFE and/or
FVCOM.
--- Initial 2004-2005 runs of ChesROMS for hypoxia with realistic physical forcing have been completed
with a 1-term temperature-dependent oxygen respiration formulation.
- Observational data of for various Chesapeake Bay parameters (e.g., S, T, TSS, O2, nutrients – ~20
parameters) at EPA monitoring stations from 1991 to 2006 have been processed and submitted to the
SURA server.
-- A software engineer at University of Boulder (Jisamma Kallumadikal) has been identified to support
“plug-and-play” interoperability of ROMs-based models with multiple grids, forcings and material flux
and fate choices.
-- Finalized date for all-hands Team Meeting at VIMS – October 1, 2010.
Compiled by Elizabeth Smith (lsmith@sura.org)
15
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Figure 1. Unstructured Grid of the Chesapeake Bay for SELFE, FVCOM
Compiled by Elizabeth Smith (lsmith@sura.org)
16
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Challenges:
-- Delays in transfer of funds to subcontracts has slowed initiation of work at Rutgers.
-- Time required to hire new staff at VIMS (post-doc beginning in October) and CSDMS (software
engineer beginning in September) has slowed progress somewhat.
Look ahead to next month
-- Post team-reviewed copy of base-line intercomparison report to SURA website.
-- In collaboration with the CI team, the NetCDF version of CH3D output for Chesapeake Bay
hydrodynamics will be transferred to LSU for SURA project on-line posting.
-- A new skill assessment metric for vertical salinity stratification is being developed based on Average
Discrete Frechet Distance (ADFD).
-- We will begin work advancing interoperability of multiple ROMS grids of Chesapeake Bay (e.g.,
CBOFS2, ChesROMS, UMCES ROMS, CHIMP) linked with multiple physical forcing data sets (e.g., winds,
riverine input) and multiple choices for material flux and fate subroutines (e.g., simplistic to more
complex oxygen models, Lagrangian tracking – c.f. oil spills).
-- Calibration of the unstructured SELFE model for Chesapeake Bay hydrodynamics will begin.
-- Initial model-data comparison will begin for MAB hydrodynamic models (e.g., ESPreSSO, HyCOM,
NCOM).
-- Continue to move forward with improved/standardized forcing (to be led by L. Lanerolle) and use
standardized forcing to run next generation of intercomparison.
-- Continue to move forward with initial model-data comparison for CB oxygen from hypoxia models
(ChesROMS, ICM, EFDC).
Compiled by Elizabeth Smith (lsmith@sura.org)
17
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
5. Coastal Inundation Modeling Progress as of 9/7/2010
(Submitted by Rick Luettich on behalf of the Testbed Inundation Team)
General
1. UNC – SURA contract in place
2. Subcontracts between UNC and UND, USF, UMassD are complete. Subcontracts between UNC and
LSU, UF nearly complete (hopefully done this week). Substantial disagreement remains in contract
language in subcontract between UNC and BIO. UNC considers BIO to be a subcontractor. Gov of Canada
considers BIO to be a collaborator. BIO is looking for a path forward.
3. SURA has secured computer time on LONI and more recently TeraGrid computers. Several PIs have
attempted to use LONI machines with partial success (Zheng – USF, Wang – VIMS, Toulany – BIO, Li –
LSU). Gary Crane indicated that only Zheng’s usage has shown up on SURA account. Requested that
others email him their login ids so he can check on their accounts.
4. Provided information on various model outputs to CI group to assist with model file conversion
efforts.
5. Inundation group has been using the testbed archive at sura.org. The CI team has set up a new
archive and will provide access information for the new archive and migration of the data on the existing
archive site to the new archive.
6. Mid-year inundation group meeting being planned in Chapel Hill, NC. Preliminary discussion (9/7/10
call) identified week of Dec 5 as good.
Skill Assessment
USACE – Progress to date
• Presented capabilities of IMEDS as a skill assessment software package – important to have well
defined methods, since often subtitles often exist in typical metrics.
• Provided a draft white paper on IMEDS capabilities as a contribution to an overall skill assessment
requirements white paper for our group.
USACE – Next Steps
• Eve Devalier will visit with Eoin Howlett at ASA this week to initiate the development of a common
NetCDF interface for IMEDS. It is anticipated that the role out of IMEDS for use by the group will wait
until after the development of the common NetCDF interface plus conversion routines from each model
output format to NetCDF.
• Eve (eve.devaliere@unc.edu) will provide IMEDS to any team member who is ready to use it. Currently
it supports ADCIRC & SWAN formatted ascii files.
• Bob Beardsley (WHOI) has agreed to take a first stab at drafting an overall skill assessment document.
USACE – Challenges
• Lack of standard model file formats
• Better definition of skill metrics required for project
Compiled by Elizabeth Smith (lsmith@sura.org)
18
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Extratropical storms
Group – Progress to date
• Scituate Harbor was chosen as the primary response focus area. This area responds strongly to
regional extratropical storms, has been identified by the area NWS Weather Forecast Offices as a high
priority area for storm surge/inundation modeling and has a high resolution grid that was developed for
the companion IOOS NECOFS project.
• April 17, 2007 “Patriot’s Day” and May 24, 2005 Nor’easter storms were selected for study because
they significantly impacted the region and they are recent enough that our current topography and
bathymetry may be valid during these times.
• July 10 – August 10, 2010 chosen as a model validation period because a water level gauge has only
recently been installed in Scituate Harbor and does not date back to 2005 or 2007.
Group – Next steps
• Agree on skill metrics for model – data and model – model inter-comparisons. Bob B. has agreed to
initiate a white paper on this for distribution to the group and discussion
Group – Challenges
• File interchangeability including structure of grid files and use of NetCDF.
UMassD/WHOI – Progress to date
• Scituate FVCOM inundation grid and basic bathy/elevation data uploaded to testbed archive
• Regional Gulf of Maine FVCOM-SWAVE grid and basic bathymetry data uploaded to testbed archive
• FVCOM v3.1 source code – code uploaded to testbed archive
• FVCOM v3.2 - FVCOM-SWAVE coupling problem found and corrected, code is currently being tested
and will be provide to the group in September.
• July 10-August 10, 2010 validation run
Forcing (tides and winds), boundary, and initial conditions for Scituate grid and FVCOM results uploaded
to testbed website together with a pdf format report for the explanation and instruction. Based on a
visual time series plot, the response appears to be primarily tidal.
• April 2007 storm
Regional Gulf of Maine grid - forcing and boundary conditions, uncoupled FVCOM and SWAVE results for
month of April 2007, pdf report including model – data comparisons uploaded to testbed website.
Scituate grid- forcing and boundary conditions, FVCOM results, pdf report uploaded to testbed website.
• May 2005 storm
Regional Gulf of Maine grid – uncoupled FVCOM and SWAVE runs completed, forcing and boundary
conditions, results will be uploaded to test bed archive this week
Scituate grid – FVCOM runs completed, forcing and boundary conditions, results will be uploaded to test
bed archive this week
• Scituate ground truth data - High water level at Ball’s house near Scituate lighthouse on a high hill at
entrance to harbor – 2005 storm: 23 ft; 2007 storm: 25.25 ft above ground level. Need method to
convert to height above local datum.
Compiled by Elizabeth Smith (lsmith@sura.org)
19
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
UMassD/WHOI – Next steps
• Try to identify a means for obtaining a datum correction for high water marks
• Continue to look for any additional data
• Re-run 2005 & 2007 storms for coupled FVCOM – SWAVE using FVCOM 3.2.
• Distribute FVCOM 3.2 to group and continue to assist LSU and USF with implementation
UMassD/WHOI – Challenges
• Observational data for skill assessment in Scituate
BIO – Progress to date
• Installing latest available version of unstructured SWAN. Can compile on LONI machine (Louie) but will
not run – MPI error. Will switch to version of unstructured SWAN that UND is running. UND has online
instructions and will help to compile/run.
• Evaluating how to convert provided FVCOM grid files (with bathy) and wind files into formats used by
WaveWatch 3 and unSWAN
• TS Earl passage provided opportunity to evaluate WW3 performance for a tropical cyclone.
BIO – Next steps
• Convert regional Gulf of Maine grid to SWAN (ADCIRC) & WW3 formats
• Convert 2007 storm wind forcing to SWAN & WW3 formats
• Run 2007 storm
BIO – Challenges
• Getting unstructured SWAN to run on multi-processor machine.
• Negotiating subcontract between BIO and UNC.
UF SLOSH – Progress to date
• Received extratropical files/code from NOAA MDL
UF SLOSH – Next steps
• Convert FVCOM wind fields (open BCs?) into format for use in SLOSH extratrop.
• Run 2005, 2007 storms
• Communicate with MDL to determine whether archived SLOSH results exist for the 2005 & 2007
storms.
UF SLOSH – Challenges
• Key person at MDL, Arthur Taylor, is on detail during the month of September and is not able to assist
with testbed activities.
UND – Progress to date
• Scituate FVCOM grid converted to ADCIRC grid. Is not stable for tidal runs using M2 forcing only from
ADCIRC tidal database. Questions about irregular bathymetry in grid and run parameters.
Compiled by Elizabeth Smith (lsmith@sura.org)
20
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
UND – Next steps
• Continue to work on getting stable tide runs on Scituate grid.
• Convert FVCOM Gulf of Maine Grids to ADCIRC format
• Convert FVCOM wind fields (open BCs) into format for use in ADCIRC.
• Run 2010 validation, 2005, 2007 storms
UND – Challenges
• Stability of Scituate grid.
VIMS – Progress to date
• Converted Scituate grid into SELFE grid.
• Performed test tidal run for Scituate for April 2007 and compared to tidal time series for that period
reconstructed from observationally based tidal harmonics.
VIMS – Next steps
• Convert FVCOM wind fields (open BCs) into format for use in SELFE.
• Run 2010 validation, 2005, 2007 storms
VIMS – Challenges
• Ability to include waves?
Compiled by Elizabeth Smith (lsmith@sura.org)
21
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Tropical storms
Group – Progress to date
• Western Louisiana, eastern Texas region selected for test bed due to extensive surge and inundation
caused by Hurricanes Rita 2005 and Ike 2008 in this region and the availability of substantial
observational data sets for model validation
• Selected Rita 2005 and Ike 2008 as study storms
• Agree to try to work with a high resolution western LA/eastern TX grid containing ~2million nodes/~4
million elements
• All groups have successfully run this grid for tides. Compute intensive, large files especially from
FVCOM which only has global output.
Group – Next steps
• Fall back to smaller grid covering only the Gulf of Mexico, ~300,000 nodes.
• Need to agree on meaningful tidal run and comparison in 2D and 3D.
• Agree on skill metrics for model – data and model – model inter-comparisons. Bob B. has agreed to
initiate a white paper on this for distribution to the group and discussion
Group – Challenges
• File interchangeability including structure of grid files. In particular, ADCIRC and FVCOM have a
different way of representing sub-grid scale structures such as levees, roadbeds, etc. – UND will convert
ADCIRC sub-grid scale structures to a form FVCOM can use.
• Size of grid makes it difficult for most participants to modify using standard grid tools (e.g., SMS) and
makes model runs infeasible on in house clusters, forcing the use of shared (SURA Grid, TeraGrid)
resources.
• Lingering difficulty getting models to run / achieving desired turnaround time on LONI shared
machines.
UND – Progress to date
• High resolution LA/TX 2 million node grid uploaded to testbed archive
• 2008 tidal boundary condition (for Hurricane Ike time period) uploaded to testbed archive
• OWI gridded 10m wind fields for Hurricane Ike uploaded to testbed archive – note, these are 30-min
averaged, marine roughness winds on a set of moving, nested grids.
• Performed initial tide and Ike ADCIRC and ADCIRC+SWAN runs, studied proper treatment of refraction
terms in SWAN
UND – Next steps
• Upload full set of observational points and data to testbed archive
• Assist other groups translate ADCIRC grid (and particularly sub-grid scale features) into FVCOM and
SELFE grids
• Provide Hurricane Ike wind fields that have been adjusted to a 10-min averaging period and modified
with ADCIRC’s land cover routine to reduce overland windspeed on 2 km regular grid
• Provide ADCIRC + SWAN results for Hurricane Ike
• Provide 300,000 node grid along with tidal BCs
Compiled by Elizabeth Smith (lsmith@sura.org)
22
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
• Provide tidal boundary condition and wind forcing for Hurricane Rita.
• Put LSU high resolution bathymetry on LA/TX 2 million node grid.
UND – Challenges
• Refraction terms in unstructured SWAN model.
UF SLOSH – Progress to date
• Ran Hurricane Ike using “out of the box” SLOSH wind model with NHC best track on two SLOSH basins
(Galveston and Sabine Pass). Provided MEOW of results.
UF SLOSH – Next steps
• Upload basin, forcing, model, results uploaded to testbed archive.
• Provide Ike time series results / maximum elevations at locations of observational data
• Developing a sequence (3-4) of nested grids to allow running of SWAN for Hurricane Ike. Requires
output and manipulation of windfields from SLOSH for use in SWAN.
UF SLOSH – Challenges
• Limited scope of SLOSH means lots of recoding/extensions to compare with other models.
LSU – Progress to date
• Uploaded recently acquired high resolution bathymetry in several important passes to testbed archive
• Successfully able to get FVCOM to compile and run on LONI machines for tides during Ike period.
LSU – Next steps
• Comparison of tidal results with observation
• Run Hurricane Ike – waiting for UND to convert ADCIRC grid(s) to FVCOM format including sub-grid
scale features and also to provide 2km Ike gridded wind fields
LSU – Challenges
• Working with 2 million node tropical grid
• Computational resources
USF – Progress to date
• Converted ADCIRC grid into FVCOM grid for tidal runs and provided revised grid to the other team
members. Did not include sub-grid scale features conversion.
• Initial evaluation of model performance (2D, 1.5s external, 15s internal time step; 96 processors = 4
hrs run time/day simulation time).
• Successfully ran tides (boundary forcing and internally forced) using FVCOM on 288 processors on
LONI and compared results to NOAA tidal gauges from FL to TX. Results uploaded to testbed archive?
USF – Next steps
Compiled by Elizabeth Smith (lsmith@sura.org)
23
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
• Additional tidal runs – 3d? May wait for 300,000 node grid
• Run Hurricane Ike – waiting for UND to convert ADCIRC grid(s) to FVCOM format including sub-grid
scale features and also to provide 2km Ike gridded wind fields
USF – Challenges
• Working with 2 million node tropical grid
• Computational resources
VIMS – Progress to date
• Converted ADCIRC grid into SELFE grid. Did not include sub-grid scale features in conversion
• Provided performance benchmarks for model runs on differing #s of processors for tidal runs (2D, 90 s
time step, 64 processors = 1 hrs run time / day simulation time).
• Ran SELFE for tides and compared results to 22 NOAA tidal gauges along North Gulf and FL shelves,
better skill found using a Manning n coefficient = 0.025 than =0.02 or 0.03. Identified two worst
comparison stations and question the need for spatially varying friction coefficients. (Does SELFE include
the internal tidal loading?)
• Ran Hurricane Ike using NCEP reanalysis wind forcing.
VIMS – Next steps
• Upload tidal results?
• Run Hurricane Ike – waiting for UND to convert ADCIRC grid(s) to FVCOM format including sub-grid
scale features and also to provide 2km Ike gridded wind fields
VIMS – Challenges
•?
Compiled by Elizabeth Smith (lsmith@sura.org)
24
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Appendix A - Super Regional Modeling Testbed Deliverables

Develop protocols for how to transition research models and skill assessment tools, from the research
environment into operational centers.

Advance a cyberinfrastructure capability that integrates tools, methodologies, and best practices to
support a modeling testbed, and includes:
o Conventions for representation of unstructured grid data in NetCDF;
o Implementation of an unstructured grid model standard in NetCDF-Java.I;
o A standard model toolkit in Matlab to perform standard operations on climate and forecast (CF)compliant OpenDAP data;
o A Web Map Service (WMS) that works with all testbed structured and unstructured grid models;
o Tools for model grid subsetting and re-gridding.

Provide a testbed that:
o Allows for rigorous, transparent and replicable testing of models;
o Uses the aforementioned cyberinfrastructure capability;
o Demonstrates the skill and performance of models in the following regions:
 Shelf hypoxia in the Northern Gulf of Mexico
 Estuarine hypoxia in Chesapeake Bay
 Storm surge and land inundation for tropical and extra-tropical storms in the
Gulf of Mexico and Gulf of Maine.

Develop and provide meaningful, quantitative and comparable analyses including sensitivity analyses, skill
scores and other skill metrics for evaluating numerical model performance. These analyses will include
model-to-model comparisons, and model-to-data comparisons. A byproduct of these analyses will be the
compilation and integration, through the cyberinfrastructure initiative, of relevant observational data (in
situ and satellite) that will heretofore be available for future model validation efforts.

Begin the transition of the most promising research model(s) and skill assessment tools from the research
environment into operational centers.

Provide skill assessment guidance and advice to the operational centers for the models that are currently
in operational use in particular with respect to model resolution, atmospheric and lateral boundary
forcing, specific physics, and model efficiency in the context of computational resources.

Develop a demonstration website to facilitate sharing of observations and model results, model testing,
and visualization of model-to-model and model-to-observations comparisons incorporating the analysis
and skill assessment tools.
Compiled by Elizabeth Smith (lsmith@sura.org)
25
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
Appendix B - TAEG comments on Testbed Team Leads meeting: Aug 31, 2010
Attendees: Denny Kirwan, Rich Signell, Frank Aikman, Yi Chao, Bruce Ebersole, Chris Mooers
Overall, the TAEG is satisfied the progress during month 2 of the IOOS Testbed, with evidence of Team
Leads overseeing continual progress toward the Year 1 high level goals. Some issues, however, need to
be addressed.
The high-level goals should reflect that established metrics for estuarine and shelf hypoxia do not
currently exist and may involve significant scientific effort to develop. This project will probably not
solve the problem, but is expected to make significant progress by coming up with some prototype
metrics and documenting the manner in which they were selected.
The TAEG is concerned that the operational side of the house is not being sufficiently engaged. We
suggest adding Gregg Jacobs (NRL), Frank Bub (NAVO), Hendrik Tolman(NOAA/NWS), and AJ Zhang
(NOAA/CO-OPS) to the TAEG to complement Bruce Ebersole (USACE) and Frank Aikman (NOAA/NOS) so
that they can better track and provide input to the program. Suggest inviting Jonathan Westcott from
FEMA to the Inundation mailing list so he can track the inundation group discussion and meetings.
Team leads are encouraged to communicate any roadblocks or problems, not just successes in their
reports.
A joint GCOOS/ SURA/OceanLeadership team for model testbed activities for the Deepwater Horizon
modeling seems like a perfect follow-on to the existing testbed.
How can we facilitate the longer term storage and distribution of data (sites like NCDDC?)
The teams should analyze preliminary findings to see if particular models or scenarios should be
dropped from further computation or analysis.
Some administrative suggestions:
-
Group reports should be available the day before (not day of) the telecon either by email or
from a web site
A standard 1-pager PPT slide should also be provided the day before the telecon to summarize
the key points and issues
Use webex ony if really necessary, given time constraints.
Cyberinfrastructure:
The codes and procedures used to convert model output to CF-compliant NetCDF (e.g. the code used to
convert Ch3D) should be documented and made available as a community resource.
IMEDS should be enhanced to access CF-compliant files or datasets via OPeNDAP. An easy way to do
this is by utilizing a NetCDF-Java based Matlab Toolkit, such as http://njtbx.sourceforge.net.
Compiled by Elizabeth Smith (lsmith@sura.org)
26
DRAFT -- SURA Monthly Progress – Super-Regional Modeling Testbed
September 10, 2010
The UAH Drupal site (http://testbed.sura.org) should feel like the one site that people need to visit to
find information about the IOOS Modeling Testbed. It should have descriptions of and links to the
services provided by the Linux server at LSU. At some point in the future it might also be a portal for
actually configuring and running models, but development of those capabilities should not take place
during this project, as this would take resources away from the other CI efforts.
Estuarine Hypoxia:
Having a software engineer working for Scott Peckham on a GUI for integrating different ROMS versions
as part of the estuarine hypoxia group seems inconsistent with the high level goals and with the
management structure. Shouldn’t this software engineer should be part of the CI group and tasked to
support the high-level goals?
We didn’t hear anything about the shelf model comparison (Wilkin’s effort to compare RTOFS,
ESPRESSO, HOPS, NYHOPS). Is that work on track?
Inundation:
The codes and procedures used to convert model grids from one model to another should be
documented and made available as a community resource. Perhaps CI can help with this.
It seems that the 2,000,000 node grid has been a burden to some groups. On the concall it was
mentioned that the 2M node grid was actually smaller than the 5M node grid that the ADCIRC team
would prefer. Could we see the assessment from ADCIRC that shows the cost/benefit of grid
refinement from several resolutions (e.g. 0.5M, 2M and 5M) to help build our intuition for how
important these grid refinement is?
Shelf Hypoxia:
What is the potential impact on the first year deliverables by Hetland not getting a postdoc?
Wiggert’s Matlab programmer should interface with CI on MAVE and PAVE development, since ASA did
a substantial part of the original development work.
Compiled by Elizabeth Smith (lsmith@sura.org)
27