Management of Oceanographic
time-series data at the Woods
Hole Coastal and Marine Science
Center
Ellyn Montgomery
USGS
NERACOOS/NECOSP Data Management Workshop,
Sept. 26, 2012
U.S. Department of the Interior
U.S. Geological Survey
USGS Coastal and Marine Geology
Program



This presentation applies only to
Oceanographic measurements acquired by
Woods Hole researchers and colleagues to
address scientific questions
The data is mostly used for validation of
sediment transport and coastal process
models
Management of the data is part of the larger
Data Life-cycle that includes everything from
planning to archival and publication
NECODP 2012
About our data



Our holdings describe conditions at specific
locations for limited durations. Except for a
25+ year program in Massachusetts Bay, we
don’t do long term monitoring programs.
Once the data files collected during an
experiment are processed and QC’d that data
is published (Federal requirement)
Updates are made only if a problem is found
in the data to correct that problem
NECODP 2012
About our data II



The data are stored in netCDF format files
using EPIC conventions
NcML is used to convert to CF conventions
Files are delivered on-line via
 file download
 OPeNDAP using THREDDS


FGDC metadata for each experiment is being
created and harvested to ocean.data.gov
Each file has ISO 19115 metadata (ncISO)
NECODP 2012
How we manage our data - 1


Our raw data is included in the WHSC field
activity tracking system
We store 3 levels of data on local disks that
are backed up nightly, with periodic backups
made and stored off-site. These levels are :
 Raw data as off-loaded from the instruments
 Processed data (and stages leading to it)
 Published data
 As EPIC
 As CF
NECODP 2012
Publication and access



Our data may be published directly to the
website after approval or with an Open File
Report that documents the context of the
measurements.
We periodically check the website for broken
links or non-responsiveness
We add new interfaces as available (for
example making KML files to use with
GoogleEarth)
NECODP 2012
Web access
NECODP 2012
Web access
NECODP 2012
THREDDS
NECODP 2012
Metadata




NetCDF allows metadata to be stored as
attributes with the data in the file
We use standards and conventions to enable
interoperability
We have GCMD DIF, FGDC and ISO19115
metadata in as many catalogs and registries
as possible
We enhance the metadata as new standards
emerge (added Unidata Dataset Discovery
terms in 2011)
NECODP 2012
Data Management Successes




Community Standards development has
evolved to enable LOT of great functionality
We are happy with how THREDDS allows
remote clients to access data for analysis in
real-time, without having to down-load files
Clients like Matlab, IDV, ncBrowse and
ncView allow access and visualization
Registering our THREDDS catalogs in larger
repositories of ocean measurements
enhances discoverability
NECODP 2012
Data Management Challenges

Getting links to our holdings into the catalogs
of the regional observatories
 Since our data is historical, it hasn’t really fit the
observatory mission, but including links to
historical measurements in the area would be
useful for enriching potential analyses


Identifying which of the emerging
technologies will persist into the future
Finding ways to let potential users know the
data is available
NECODP 2012
Take home ideas




Use standards to enhance discoverability and
enable use of community developed tools
Enable web-based access so the users can
access what they need without assistance
Tailor the process to what your clients need
to accomplish their missions
Adapt your data and services to take
advantage of new techniques as they evolve
NECODP 2012
Web access



The human-friendly interface to our
experimental data, (EPIC conventions) is at:
http://stellwagen.er.usgs.gov
The computer-friendly (CF conventions)
version of our data holdings, served by
THREDDS at:
http://geoport.whoi.edu/thredds/allts_reg_cat
alog.html
Contact me at: emontgomery@usgs.gov
 (508)457-2356
NECODP 2012