The HDF Group
Easy Access of NASA HDF
data via OPeNDAP
Kent Yang and Joe Lee
The HDF Group
September 28,2010
HDF/HDF-EOS Workshop XIV
1
www.hdfgroup.org
Easy
September 28,2010
HDF/HDF-EOS Workshop XIV
2
www.hdfgroup.org
Outline
• Introduction to OPeNDAP
• HDF4/5-OPeNDAP Handlers
• Live Demo
September 28,2010
HDF/HDF-EOS Workshop XIV
3
www.hdfgroup.org
OPeNDAP
• A software framework that allows simple access
to remote data
• Client-server model
• Servers make the local data accessible remotely
• Clients make applications easily access to remote
served data
• Widely used by Earth Science Community
• URL: http://www.opendap.org/
September 28,2010
HDF/HDF-EOS Workshop XIV
4
www.hdfgroup.org
Access HDF via FTP
Aqua/Aura
Users
IDL
AIRS/OMI
FTP
Local HDF
Files
September 28,2010
FTP
Clients
FTP
Servers
HDF/HDF-EOS Workshop XIV
HDF Files
5
www.hdfgroup.org
OPeNDAP
Concept
Typical
Remote
Access
View Data
(User)
FTP/HTTP
Remote Data
(HDF4/5)
Visualization
tools
Visualization
(e.g.,
grads)
Tools
(gradsdap)
Handler
(hdf4/5_handler)
Client Library
(libnc_dap)
Server
(Hyrax)
Data Access Protocol (DAP) via Internet
September 28,2010
HDF/HDF-EOS Workshop XIV
6
www.hdfgroup.org
Access HDF via OPeNDAP
Aqua/Aura
Users
IDV
Hyrax
HDF HANDLER
AIRS/OMI
libnc-dap
DAP
Visualization
Tools
September 28,2010
OPeNDAP
Clients
OPeNDAP
Servers
HDF/HDF-EOS Workshop XIV
HDF Files
7
www.hdfgroup.org
OPeNDAP Data Types
• OPeNDAP provides three basic data objects:
- data descriptor structure (.dds)
syntactic metadata – rigid
- data attribute structure (.das)
semantic metadata – flexible
- data (.dods) - the actual data in a binary structure
• Data are requested with a URL
September 28,2010
HDF/HDF-EOS Workshop XIV
8
www.hdfgroup.org
OPeNDAP Servers
CODAR
netCDF HDF4/5 Matlab
Data
Data
CODAR
netCDF
Data
Tables
SQL
FITS
CDF
Data
Data
Data
Data
Data
Data
Matlab
HDF4/5
September 28,2010
Data
DSP
Flat
Binary
JGOFS
DSP
FITS
JDBC
HDF/HDF-EOS Workshop XIV
CEDAR
General
Data
Data
ESML
FreeFrom
CDF
CEDAR
9
www.hdfgroup.org
OPeNDAP Clients
netCDF C
Ferret
GrADS
September 28,2010
netCDF Java
IDV
VisAD
ncBrowse
Matlab
Client
IDL
Client
Matlab
IDL
HDF/HDF-EOS Workshop XIV
Access
10
Excel
www.hdfgroup.org
Outline
• Introduction to OPeNDAP
• HDF4/5-OPeNDAP Handlers
• Live Demo
September 28,2010
HDF/HDF-EOS Workshop XIV
11
www.hdfgroup.org
Enhanced HDF4 Handler
Enhanced HDF4 HANDLER
Aura
Users
IDV
MODIS Swath
w/ Dim. Map
Hyrax
libnc-dap
DAP
Visualization
Tools
September 28,2010
OPeNDAP
Clients
OPeNDAP
Servers
HDF/HDF-EOS Workshop XIV
HDF4 Files
12
www.hdfgroup.org
An example: Access HDF data via OPeNDAP
Swath with Dimension Map
September 28,2010
HDF/HDF-EOS Workshop XIV
13
www.hdfgroup.org
HDF-EOS2 Products Supported
• GES DISC
• AIRS / MERRA / TOMS
• LAADS / LP DAAC / NSIDC
• Many MODIS products
• NSIDC
• AMSR_E / NISE
• LaRC
• MISR / MOPITT / some CERES(TRMM)
September 28,2010
HDF/HDF-EOS Workshop XIV
14
www.hdfgroup.org
Pure HDF4 Products Supported
• GES DISC
• TRMM L1B, L2B Swath
• TRMM L3 Grid 42B & 43B products
• OBPG (Ocean Color)
• SeaWiFS / MODIST / MODISA
• CZCS / OCTS
September 28,2010
HDF/HDF-EOS Workshop XIV
15
www.hdfgroup.org
Pure HDF4 Products Supported
• CERES
•
•
•
•
•
•
•
September 28,2010
CER_AVG
CER_ES4
CER_ISCCP-D2like-Day
CER_ISCCP-D2like-GEO
CER_SRBAVG
CER_SYN
CER_ZAVG
HDF/HDF-EOS Workshop XIV
16
www.hdfgroup.org
Limitations
• Reasons for Visualization Failures
• IDV / Panoply / Java Heap Memory
• Network / DAP (e.g., max HTTP header size)
• Can’t retrieve key information correctly (e.g.,
LAMAZ projection) from data
• Unable to harmonize with CF-conventions (e.g.,
FillValue inside latitude and longitude)
September 28,2010
HDF/HDF-EOS Workshop XIV
17
www.hdfgroup.org
Limitations(continued)
• HDF-EOS2 + additional HDF4 objects
• Additional HDF4 objects will be ignored.
• The rest of pure HDF4 products not
listed in previous slides are not tested
and may not be visualized by Java
OPeNDAP Clients.
September 28,2010
HDF/HDF-EOS Workshop XIV
18
www.hdfgroup.org
HDF-EOS5 products Supported
• OMI Swath, Grid
• MLS/HIRDLS Swath
• TES Grid
September 28,2010
HDF/HDF-EOS Workshop XIV
19
www.hdfgroup.org
Learning Curve of accessing HDF data
NCL
From the ESIP wiki page:
http://wiki.esipfed.org/index.php/Making_Science_Data_Easier_to_Use_with_OPeNDAP
Making Science Data Easier to Use with OPeNDAP
September 28,2010
HDF/HDF-EOS Workshop XIV
20
www.hdfgroup.org
Live Demo
September 28,2010
HDF/HDF-EOS Workshop XIV
21
www.hdfgroup.org
OPeNDAP HDF4/HDF5 handlers
• Source codes and RPMs can be found under
http://opendap.org
• Demo HDF5/HDF4 servers
http://hdfeos.org/zoo/hdf5_handler/
http://hdfeos.org/zoo/hdf5_handler/
• Currently NASA GES DISC has installed both
OPeNDAP HDF4 and HDF5 handlers
September 28,2010
HDF/HDF-EOS Workshop XIV
22
www.hdfgroup.org
The HDF Group
Thank you !
September 28,2010
HDF/HDF-EOS Workshop XIV
23
www.hdfgroup.org
Acknowledgements
This work was supported by cooperative agreement
number NNX08AO77A from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the author[s] and do not necessarily reflect
the views of the National Aeronautics and Space
Administration.
September 28,2010
HDF/HDF-EOS Workshop XIV
24
www.hdfgroup.org
The HDF Group
Questions/comments?
September 28,2010
HDF/HDF-EOS Workshop XIV
25
www.hdfgroup.org
Our Work
• Let Visualization Clients Access HDFEOS2 and HDF4 files
September 28,2010
HDF/HDF-EOS Workshop XIV
26
www.hdfgroup.org
HDF5-OPeNDAP Handler Update
• Support Aura HDF-EOS5 OMI Swath
• Access some files that follow HDF5
Dimension Scale Specification
• Support Aura HDF-EOS5 MLS
September 28,2010
HDF/HDF-EOS Workshop XIV
27
www.hdfgroup.org