Weathertop Consulting, LLC
Server-side OPeNDAP Analysis - A
General Approach Utilizing
Legacy Applications through TDS
Roland Schweitzer
Weathertop Consulting, LLC
Steve Hankin and Ansley Manke
NOAA/PMEL
Highlights

Server-side analysis





Motivation
LAS as OPeNDAP Client and Server
Evolution of the server implementation
Community “call to action”
Summary
Weathertop Consulting, LLC
Server-side Analysis


In general server-side analysis is a computation
made by an OPeNDAP server at the request of
a client.
The specification of the computation is
transmitted to the server via the OPeNDAP
URL.
Weathertop Consulting, LLC
Motivation

We are interested in server-side analysis for use with the
Live Access Server (LAS).



Primarily as a way to implement comparisons between data
defined on different grids
We want our implementation to leverage the analysis
capabilities of legacy applications like Ferret and
GrADS.
We want to use our experience running legacy
applications (like Ferret) from within a Java runtime
environment.
Weathertop Consulting, LLC
The Live Access Server (LAS)
A highly configurable Web server designed to provide
flexible access to geo-referenced scientific data
Weathertop Consulting, LLC
LAS Product Server
Product
Server
Metadata
(XML)
client
SQL
Backend
Service
Ferret
Backend Service
Ferret
product
Local
netCDF
data
Local
RDBMS
metadata
product request
XML (REST)
back end
request (SOAP)
DRDS
Backend
Service
OPeNDAP
server
DRDS
server
netCDF
data
Remote
RDBMS
LAS Architecture
Weathertop Consulting, LLC
LAS
Product
Server
Metadata
(XML)
user
SQL
Backend
Service
Ferret
Backend Service
Ferret
DRDS
Backend
Service
product
Suppose the variables
are on different grids?
netCDF
data 1
metadata
product request
XML (REST)
back end
request (SOAP)
OPeNDAP
server
OPeNDAP
server
netCDF
data 2
Comparing OPeNDAP
datasets
Weathertop Consulting, LLC
LAS as an OPeNDAP Server



Data on grids which are available via LAS are
guaranteed to be geo-referenced and at least
COARDS compliant.
We can often “repair” (including re-gridding) the
data and/or metadata by associating a script of
Ferret commands with the data source in the
LAS configuration.
Wouldn’t it be nice to make these “repaired”
data available via OPeNDAP?
Weathertop Consulting, LLC
The Ferret Data Server



FDS made this possible.
FDS provides an OPeNDAP view of the data
being served by LAS and makes any
transformations specified by the associated
script before serving the data.
FDS also implements server-side analysis
(including the ability to pass in external data
sources).
Weathertop Consulting, LLC
A GDS Digression



The GrADS Data (DODS) Server is the first
implementation of this concept.
In fact, FDS used the Anagram framework upon
which GDS is built.
Both GDS and FDS use the Java Runtime
environment to invoke the associated legacy app
(GrADS or Ferret) to do the heavy lifting.
Weathertop Consulting, LLC
FDS Capabilities

FDS took advantage of several characteristics of the
underlying engine (Ferret).




New "virtual" data variables can be defined
Can build the metadata (netCDF header described by
dimensions, coordinate variables and the structure of data
variables) without performing any heavy calculations for both
data read from files and “virtual” data variables
Only performs calculations when the data are requested
Only calculates the minimal set needed to fulfill the current
request
Weathertop Consulting, LLC
FDS Evolution




Keep these advantages and evolve the implementation.
The Java netCDF library allows new data container
formats to be plugged-in by implementing the I/O
Service Provider interface.
Once “plugged-in” clients using nj22 have access to the
data from this container.
We implemented a Ferret I/O Service provider which
can read Ferret command scripts and direct Ferret to
perform the calculations as needed to satisfy data
requests.
Weathertop Consulting, LLC
Example
% more data/simple4.jnl
use levitus_climatology
let/d=levitus_climatology temp_20 =temp[d=levitus_climatology,z=0:20@sum]
set var/title="surface heat content"/units="deg C“ temp_20[d=levitus_climatology]
Weathertop Consulting, LLC
Example
dncdump -c http://porter.pmel.noaa.gov:8920/thredds/dodsC/mydata/simple4.jnl
float TEMP(ZAXLEVITR, YAXLEVITR, XAXLEVITR) ;
TEMP:units = "DEG C" ;
TEMP:long_name = "TEMPERATURE" ;
TEMP:_FillValue = -1.e+10f ;
TEMP:missing_value = -1.e+10f ;
TEMP:dataset = "levitus_climatology.cdf" ;
float TEMP_20(YAXLEVITR, XAXLEVITR) ;
TEMP_20:dataset = "levitus_climatology" ;
TEMP_20:direction = "IJ" ;
TEMP_20:units = "deg C" ;
TEMP_20:long_name = "surface heat content" ;
TEMP_20:missing_value = "-1.00000E+34" ;
TEMP_20:virtual = "true" ;
Weathertop Consulting, LLC
IOSP*
Application
NetcdfFile
The low-level
part of the
NetCDF-Java
version 2.2
architecture
I/O service provider
NetCDF-3
NetCDF-4
HDF5
NIDS
isValid
GRIB
open
GINI
Nexrad
readData
DMSP
Ferret
Java Runtime
GrADS
*Stolen directly from John Caron with only this measly acknowledgement.
Weathertop Consulting, LLC
The THREDDS Data Server



TDS is an OPeNDAP server.
nj22 I/O Service Providers can be plugged in to
TDS.
The combination of the Ferret I/O Service
Provider and TDS (aka F-TDS) serves via
OPeNDAP data which are represented by Ferret
command scripts (both data read from disk by
Ferret and virtual data computed on-the-fly by
Ferret).
Weathertop Consulting, LLC
F-TDS and Server-Side Analysis




A DataSource handler can also be plugged in to TDS
which allows custom handling of OPeNDAP requests
based on the contents of the HTTPServletRequest
Object (and by implication the URL).
We built such a DataSource handler which recognizes
URL with embedded analysis expressions.
The three groups of input to the server-side analysis
were copied from GDS.
The three sets are:



Data sources (e.g. OPeNDAP URLs).
Analysis “commands” which are implementation specific.
A sub-region.
Weathertop Consulting, LLC
An Analysis URL
http://machine:port/thredds/dodsC/
_expr_
{dataset1,dataset2,...}
{expression1;expression2;...}
Part of the original GDS
specification, not necessary
{region}
and not often used with FDS
.URLsuffix?constraint
Weathertop Consulting, LLC
FDS-Specific Example
http://host.gov:9090/thredds/dodsC/data/coads
_expr_
{http://host.gov:9090/thredds/dodsC/data/levitus}
{DIF=SST[d=1]-TEMP[d=2,g=SST[d=1]]}
.asc?DIF
Weathertop Consulting, LLC
Community Action

Can we define a implementation-indepedent
syntax for server-side analysis requests?
Some common operations (averaging, differencing,
linear interpolation) with standard name and syntax
 Server-specific (native) operations


What mechanism?

An encoded XML string?


We use this technique in LAS with good success
A simple command language?
Weathertop Consulting, LLC
XML Analysis Expression
<datasets>
<dataset id=1>"my_local_dset.nc"</dataset>
<dataset id=2>"http://remote_dset.nc"</dataset>
</datasets>
<operation>
<name>DIFF</name>
<arg pos=1 dset=1>sst</arg>
<arg pos=2 dset=2>sst</arg>
</operation>
Weathertop Consulting, LLC
With “native” Operation
<datasets>
<dataset id=1>my_local_dset.nc</dataset>
<dataset id=2>http://remote_dset.nc</dataset>
</datasets>
<native_operation>
LET diff = sst[d=2]-sst[d=1]
</native_operation>
Weathertop Consulting, LLC
Summary



Server-side analysis is critical for the future LAS.
NetCDF Java 2.2 and the THREDDS Data
Server is a great platform for implementing this
type of analysis with a legacy analysis application
(like Ferret).
A community-developed server-side analysis
framework would make it easier to get the
advantages of server-side analysis from other
servers.
Weathertop Consulting, LLC
Acknowledgments

From COLA




Jennifer Adams
Brian Doty
Joe Wielgosz

Former TMAP


Richard Rogers
Yonghua Wei
From Unidata


John Caron
Ethan Davis
Weathertop Consulting, LLC