RAINEX Planning Meeting, Miami, FL
RAINEX Data Management
UCAR Joint Office for Science Support
José Meitín
Jim Moore
Dick Dirks
RAINEX Planning Meeting, Miami, FL
JOSS Data Management
•Early involvement in project planning
•Involvement with PIs to develop data management
strategy
•Consistent implementation of strategy for lifetime of
project and beyond
•Reliable and efficient archive and distribution
system
•Easy and efficient access to datasets by broader
community
UCAR Office Of Programs
Joint Office for Science Support
RAINEX Planning Meeting, Miami, FL
Data Management Considerations
•Develop a consistent Data Policy
•Consider Data Types Formats and Documentation (metadata)
•Data Collection strategies (operations support, supplementary data)
•Real-time Requirements ( data calibration, intercomparison, and
needs for operational decision making)
•Data Quality Control and Processing responsibilities
•Data Archive structure, functions and responsibilities
•Data Distribution protocols (user interface, metadata inventories,
browse products)
•Collaboration with other programs and data centers (interoperability)
RAINEX Planning Meeting, Miami, FL
RAINEX Planning Meeting, Miami, FL
JOSS Data Services
•
•
•
•
•
•
•
•
Data Questionnaire
Data Management Plan
Project Web Pages
Real-time Operations and Data Ingest
Field Data Catalog
Quality control and dataset development
Interactive Data Archive and Distribution
Special media products (e.g. CD-Roms)
UCAR Office Of Programs
Joint Office for Science Support
RAINEX Planning Meeting, Miami, FL
JOSS Data Services
Data Questionnaire
UCAR Office Of Programs
Joint Office for Science Support
RAINEX Planning Meeting, Miami, FL
JOSS Data Services
•
Project Web Pages
RAINEX Planning Meeting, Miami, FL
JOSS Data Services
•
Real-time Operations Support and Data Ingest
Satellite Ground Station
Forecast Products
Numerical Model Analyses
Operational Data Streams
UCAR Office Of Programs
Joint Office for Science Support
RAINEX Planning Meeting, Miami, FL
GTS METAR Observation Locations
6-10 November 2004
RAINEX Planning Meeting, Miami, FL
GTS Ship and Buoy Observation Locations
6-10 November 2004
RAINEX Planning Meeting, Miami, FL
Coastal and Marine Observation Locations
*
*
o
o
NDBC Buoy
CMAN
NWLON
PORTS
o
*
+
o
+
TCOON (TX)
TABS (TX)
LUMCON (LA)
WAVCIS (LA)
COMPS (FL)
RAINEX Planning Meeting, Miami, FL
Land Observation Locations
*
*
*
o
+
ASOS
AWOS
AWOS modem
GAEMN (GA)
GFC (GA)
o
+
o
+
o
LAIS (LA)
TX ET (TX)
FAWN (FL)
TNRCC (TX)
ALMESO (AL)
RAINEX Planning Meeting, Miami, FL
JOSS Data Services
• Field Data Catalog (WWW based)
Daily Mission Reports
Operations Summary
Status Reports
Data Analysis Products
Authoring Tools
UCAR Office Of Programs
Joint Office for Science Support
RAINEX Planning Meeting, Miami, FL
JOSS Data Services
• Quality control and dataset development
Processing of research-quality, high resolution data
Processing of standard operational data streams
Compatible formats
Composite datasets and data integration
UCAR Office Of Programs
Joint Office for Science Support
RAINEX Planning Meeting, Miami, FL
Composite Data Sets at UCAR/JOSS
A composite dataset is a collection (over some time period
and region) of similar data (e.g. surface meteorological)
from a variety of sources, put into a common format, and
passed through a uniform quality control.
Why does UCAR/JOSS develop composites?
- Provides data in a uniform format with QC.
- Allows determination of network/site problems.
- Useful for model applications.
- Prevents duplication of effort.
RAINEX Planning Meeting, Miami, FL
KVII
CVT
ARM
CVT
LTER
CVT
AWOS
CVT
HRLY
Visual
ASOS
CVT
HRLY
Visual
ABLE
CVT
HRLY
Visual
HPCN
CVT
Visual
SCAN
CVT
Visual
NMSU
CVT
Visual
GWMD
CVT
Visual
MADIS
CVT
Visual
Ameriflux
CVT
Visual
NCAR (4)
CVT
HRLY
Visual
OKMESO
CVT
HRLY
Visual
WTXMESO
CVT
HRLY
Visual
Others (7)
CVT
Visual
HRLY
Visual
Visual
Visual
IHOP_2002 Hourly Surface
Composite Development
Gross Limit
Checks
MERGE
Horizontal
QC
Examine
Statistics
Visual
Hourly
Surface Composite
RAINEX Planning Meeting, Miami, FL
Aircraft Flight Level Datasets
NCAR / RAF
NetCDF
NOAA / AOC
“standard” format
JOSS RAINEX
Archive
Integrated RAINEX NetCDF
JOSS convert
“type 5” to
NetCDF
RAINEX Planning Meeting, Miami, FL
JOSS DATA MANAGEMENT TOOLS
Field Catalog
In-field tool to ingest and display
operational and preliminary research
data and project documentation for
making real-time decisions and
evaluating project progress
Features:
• Daily Mission Reports
• Operations Summary
Interactive Data Archive/Access
System
Primary means for all project
scientists and researchers to
browse and retrieve data from any
JOSS-supported projects
Features:
• Long-term field project data
archival and distribution
• Facility Status Reports
• Interactive data browsing,
subsetting, and format conversion
• Data Analysis Products
• Remote Archive Access
• Authoring Tools
• Value-added datasets
• Web-based access
• Data documentation
RAINEX Planning Meeting, Miami, FL
JOSS Data Services
http://www.joss.ucar.edu/codiac/
• Interactive Data Archive and Distribution
Ease of Data Access (web based)
Data available via FTP, magnetic tape, CD-Rom
Metadata descriptions
Data Browse, Subsetting, Format Translation Capabilities
Quality Control
Distributed Data Archives
Customized data ordering and reformat
UCAR Office Of Programs
Joint Office for Science Support
RAINEX Planning Meeting, Miami, FL
RAINEX Data Management Issues:
•Data Policy
•Data Management Plan
•Real Time Data Needs (model, satellite)
•Distributed Archive Responsibilities
•Special Data Processing (surface, upper-air, airborne
radar aircraft flight-level)
RAINEX Planning Meeting, Miami, FL
http://www.joss.ucar.edu/
meitin@ucar.edu
jmoore@ucar.edu
dirks2@ucar.edu
UCAR Office Of Programs
Joint Office for Science Support
Acronyms
PORTS – Physical Oceanographic Real-Time System
NWLON – National Water Level Observation Network
TCOON – Texas Coastal Ocean Observation Network (TAMU – Corpus)
TABS – Texas Automated Buoy System (TAMU)
LUMCON – Louisiana Universities Marine Consortium
WAVCIS – Wave Current Surge Information System (LSU)
COMPS – Coastal Ocean Monitoring and Prediction System (USF)
GAEMN – Georgia Automated Environmental Monitoring Network (U of Georgia)
GFC – Georgia Forestry Commission
ALMESO – Alabama Mesonet (Auburn U)
LAIS – Louisiana Agriclimatic Information System (LSU)
TX ET – Texas Evapotranspiration Network (TAMU)
FAWN – Florida Automated Weather Network (U of Florida)
TNRCC – Texas Natural Resources Conservation Commission