Evolution status and plans for NSIDC
DAAC EOS Core System (ECS)
Vince Troisi
1
PoDAG 25: Data Systems
Evolution status and plans for ECS:
EOSDIS Evolution 2015 Vision Tenets
Vision Tenet
Archive Management
EOS Data
Interoperability
Future Data Access
and Processing
Data Pedigree
Cost Control
User Community
Support
IT Currency
Vision 2015 Goals
 NASA will ensure safe stewardship of the data through its lifetime.
 The EOS archive holdings are regularly peer reviewed for scientific merit.
 Multiple data and metadata streams can be seamlessly combined.
 Research and value added communities use EOS data interoperably with other relevant data
and systems.
 Processing and data are mobile.
 Data access latency is no longer an impediment.
 Physical location of data storage is irrelevant.
 Finding data is based on common search engines.
 Services invoked by machine-machine interfaces.
 Custom processing provides only the data needed, the way needed.
 Open interfaces and best practice standard protocols universally employed.
 Mechanisms to collect and preserve the pedigree of derived data products are readily
available.
 Data systems evolve into components that allow a fine-grained control over cost drivers.
 Expert knowledge is readily accessible to enable researchers to understand and use the data.
 Community feedback directly to those responsible for a given system element.
 Access to all EOS data through services at least as rich as any contemporary science
information system.
Feb 3, 2005
Evolution status and plans for ECS:
Technology Infrastructure
• Commodity Servers
 IBM Blade Servers
o
Redhat Linux OS
 SGI ALTIX 350
o
Suse Linux OS, Sybase/SQS, FC/SCSI Disk Array
• Enhanced storage system to support Data Pool
ingest, archive disk cache, and Data Pool storage
 Fiber Channel SAN (SATA)
o
EMC CX700 disk array (74 TB SATA)
 ADIC/Quantum StorNext File System
• To extent hardware allows, all data on rotating disk
 ADIC/Quantum StorNext Storage Manager replaces AMASS
for providing file system for data stored in tape silo
3
PoDAG 25: Data Systems
Evolution status and plans for ECS:
Technology Infrastructure (HW Architecture Transition)
Local Raid
Local
Monitor
Enterprise
Server
Local Raid
Local Raid
Enterprise
Server
Enterprise
Server
Processing
Blades
SAN
Local Raid
Enterprise
Server
Data Pool
Storage
GigE
Switch
Local Raid
Enterprise
Server
Enterprise
Server
Local Raid
Enterprise
Server
Local Raid
FUTURE:
 Rack mounted blades
 Linux OS
 Rack mounted SAN
 Optical fiber network infrastructure
 All platforms connected to SAN
TODAY:
 Server platforms with local RAID
 Solaris or Irix OS
 Switched GigE Network
 Some platforms connected to SAN
4
PoDAG 25: Data Systems
Evolution status and plans for ECS:
Technology Infrastructure (Target HW Architecture)
RAID/SAN
Fibre
Channel
Switch
Commodity
Blade
Server
Commodity Blade Server
Silos/Drives
Commodity-Based
Shared Memory
DB Server
Commodity
Shared
Memory
DB Server
Shared SAN with
Shared Filesystem
5
PoDAG 25: Data Systems
Evolution status and plans for ECS:
Technology Infrastructure (Software Transition)
•
ECHO to provide user search, browse, and order submission functions
to the evolved ECS
 Warehouse Inventory Search Tool (WIST) and other ECHO clients will provide
these services
 The V0Gateway will be removed
 ECS orders will be implemented by ECHO clients using SOAP messages (via
HTTP? JMS?) to establish the ECHO WSDL Ordering Component (EWOC)
•
Data Pool drill down search interface would be retained
 Java application operating in the Tomcat Java Application Server
•
Other services retained include HDF-EOS to GeoTiff format conversion
utility (HEG) and the HEW Subsetter Appliance
•
Machine to Machine Gateway will be removed
 Still assessing the impact if this capability is removed from ECS
 Primary use of this service is by NSIDC staff
 May collaborate with LP DAAC (EDC) by providing requirements for a MTMGW
service via the ECHO client
6
PoDAG 25: Data Systems
Evolution status and plans for ECS:
Technology Infrastructure (Current SW Architecture)
Request Flow
Data Flow
Users
Ingest
Requests
Searches &
Orders
Search & Order
Gateways
Ingest Services
Metadata
Inserts
Searches
Results
Metadata
Data Services
Metadata
Events
Insert
Requests
Processing
Services
Distribution
Requests
Email
Notifications
Distribution
Requests
Subscription
Services
Distribution
Requests
Distribution
Services
Data
Insert
Requests
Recall
Requests
Data Pool
Services
Browse
Searches
Downloads
Storage Management Services
FSMS
Metadata
…
FSMS
SAN
Disk Caches & Nearline
Tape Storage
7
PoDAG 25: Data Systems
Users
Data Providers
Evolution status and plans for ECS:
Technology Infrastructure (Target SW Architecture)
Users
ECHO
DN,
Granules,
(ftp, scp,
e-mail)
Orders
(tcp/ip)
Processing
Requests
DAAC
On-Demand
Processing
Data Providers
PDR,
Granules
(ftp, scp)
Order
Management
Service
Data Pool
Service
Request
Staging
Processing
Requests
Subsetting
Reformatting
HEG
Service
Data Pool SAN
(StorNext) Inventory
DB
Anonymous
FTP Service
Direct Downloads
Bulk data flows
Control flows, messages
8
PDRD,
PAN QA Updates
(ftp, scp, (e-mail)
e-mail)
Metadata
(ftp)
Granules
Metadata
(ftp)
Tape
Archive
(StorNext)
Data Pool
Web Application
Metadata
(http)
Users
PoDAG 25: Data Systems
Discovery (Data Pool drill down)
Data orders
On-demand processing
Order status inquiries
Status and Plans for NSIDC DAAC
non-ECS Systems
Vince Troisi
9
PoDAG 25: Data Systems
Status and Plans for non-ECS Systems:
Technology Infrastructure (Hardware Architecture)
•
Commodity Servers
 Intel and AMD Servers with Linux OS
o
Phase two of transition in progress includes transition of data production software to
AMD servers
 Intel Servers with Windows OS
•
Enhanced Storage System to support NSIDC Storage Infrastructure
 Fiber Channel SAN (SATA and SCSI)
o
~14 TB including 3 TB to enable GLAS subsetting
 ADIC/Quantum StorNext File System (in progress)
o
Evaluating ADIC/Quantum StorNext Storage Manager as a replacement for AMASS
o
Working issue related with certification of SNSM for Copan MAID vtl
o
May need to re-evaluate the Copan MAID vtl as an archive solution
o
•
Considering disk archive solutions
Network upgrade for servers to Gigabit Ethernet
 Switchover is scheduled for October
10
PoDAG 25: Data Systems
Status and Plans for non-ECS Systems:
Technology Infrastructure (Current Capabilities)
•
Metadata Database



•
•
11
o
DIFs to GCMD and to NSIDC dynamic Catalog pages
o
Transition of CIMS (V0 Database Server) planned
TRAK
Data Operations and Preservation Metadata application
Production Software

•
Writer’s interface and Publisher via Cata-matic
Grid-o-matic
Tools

Passive Microwave Swath Data Tools

AMSR Swath to Grid Tool

MODIS Swath to Grid Tool

EASE-Grid Geolocation Toolkit

NSIDC GLAS Altimetry Extraction Tool
Search and Order Interface

GISMO (Geographic Interface for Subsetting, Mapping, and Ordering)

SNOWI (Search and Order Web Interface)

EOS Data Gateway (EDG, developed externally)
PoDAG 25: Data Systems
Status and Plans for non-ECS Systems:
Technology Infrastructure (Current Capabilities)
• Development Tools
 Mapx
o A library for mapping and gridding transformation
 JAZ
o A JAVA geographic search interface for selecting an area of
interest on a map
o Basic geographic query function in GISMO and PSQ
o Infused into the spatial search functionality of the EDG
 Backtrack
o The Backtrack Orbit Search Algorithm is a unique spatial
search method for orbital data. Backtrack has a degree of
accuracy that rivals predict methods while being faster, less
costly to implement, and less costly to maintain than any other
method
o In process of generating an RFC to establish Backtrack as a
standard orbital search algorithm within the Process and
Standards subgroup of the NASA Earth Science Data Systems
Working Group
o Has been integrated into ECHO for searching orbital data
collections
12
PoDAG 25: Data Systems
Status and Plans for non-ECS Systems:
Recent technology infrastructure evolution activities
• Deployment of LDAP
• Installation of StorNext File System for establishing a FC SAN
 Completion of installation scheduled for October
• Established a SOAP Server environment
 Used to provide database service for administrating checksums of files
stored in the archive
• Deployment of MapServer
 MODIS Mosaic of Antarctica and RAMP
 Atlas of the Cryosphere (under development)
• Additional Web services
 Export MDDB Catalog information to Google base
 Export of location information from MDDB Catalog into KML format to
Google Earth
 Sea Ice Time Series in KML (Google Earth)
13
PoDAG 25: Data Systems
Status and Plans for non-ECS Systems:
Recent technology infrastructure evolution activities
• Added Postgresql and PostGIS to development environment
 Evaluating this technology for use as a geospatial database in lieu of
Sybase
 Installed Tomcat to evaluate as a JAVA Application Server
solution
 Installed GeoServer in development environment for evaluation
 In process of re-engineering GISMO
 Adding option for Browse of subsetted data via Google Earth
 Expose backend processes as public web services
• Developed utility to convert binary to GeoTiff for our Polar
Stereographic and EASE gridded data sets
14
PoDAG 25: Data Systems
BACKUP SLIDES
15
PoDAG 25: Data Systems
Capabilities Envisioned for Future Science Information Systems
The Technology Infusion Working Group, one of four working
groups established by the NASA-sponsored Earth Science Data
Systems Working Group (aka SEEDS), identified 10 capabilities
in their vision of what information systems must provide to
support scientific research in the future. This activity was
conducted in parallel with the formulation of the NASA EOSDIS
Evolution Vision for 2015.
The group is in the process of assessing the correlation
between their top ten capabilities and the goals identified in the
NASA EOSDIS Evolution Vision for 2015.
16
PoDAG 25: Data Systems
Future Science Information Systems: Top Ten Capabilities
Scalable
Analysis
Portals
Community
Modeling
Frameworks
Interactive
Seamless
Data Analysis Data Access
Assisted Data
& Service
Discovery
Interoperable Responsive
Information Information
Services
Delivery
Evolvable Technical Infrastructure
17
PoDAG 25: Data Systems
Assisted
Knowledge
Building
Verifiable
Information
Quality
Capabilities Vision: Enabling Technologies
•
Scalable Analysis Portals






•
Assisted Data and Service Discovery



•
Network data access protocols (OpenDAP, WMS/WCS, WebDAV, GridFTP)
Established data server tools (MapServer, DODS/LAS, ArcWeb)
Interoperable Information Services


18
Visual grammars
Visual programming environments (Cantata, Triana, Viper)
High-level analysis tools (IDL, Matlab, Mathematica)
Seamless Data Access


•
Data and service description standards (XML, WSDL, RDF, OWL, DAML), web service
directories (UDDI), syndication services (RSS), topic maps
Rule-based logic systems
Established directory services (GCMD, ECHO, THREDDS)
Interactive Data Analysis



•
Assisted data & service discovery
Interactive data analysis
Seamless data access
Interoperable information services
Responsive information delivery
Verifiable information quality
Network service protocols (SOAP, Java Remote Method Invocation, OpenDAP, WS-*)
Grid toolkits (Globus)
PoDAG 25: Data Systems
Capabilities Vision: Enabling Technologies (Continued)
•
Assisted Knowledge Building



•
Community Modeling Frameworks


•
Optical networks (National LambdaRail)
Peer-to-peer networks with swarming (Modster)
Direct downlink of satellite observations
Evolvable Technical Infrastructure



19
Data pedigree algorithms (NEXRAD Data Quality Control Algorithms)
Machine-readable formats (XML)
Responsive Information Delivery



•
Multi-model frameworks (ESMF, Model Coupling Toolkit, COCOLIB)
Model data exchange standards (BUFR, GRIB)
Verifiable Information Quality


•
Data mining algorithms (Support vector machines, independent component analysis, rule
induction)
Data mining toolkits (Adam, D2K, Darwin)
Data mining plug-ins (IMAGINE, ENVI, ArcGIS)
Processor & storage virtualization software (VMware, volume manager)
Scalable architectures (Beowolf, Grid)
Bandwidth-on-demand
PoDAG 25: Data Systems
Correlation Matrix: Capabilities Vision and Vision 2015 Goals
Scalable
Analysis
Portals
Community
Modeling
Framework
Assisted
data &
service
discovery
Assisted
Knowledge
Building
Interactive
data
analysis
Seamless
data access
Interoperabl
e
information
services
Responsive
information
delivery
Evolvable
Verifiable
Technical
information
Infrastructur
quality
e
Safe Stewardship
1
Peer Review
3
Combining data
streams
seamlessly
EOS
interoperability
with other
Mobile processing
and data
3
2
1
3
2
2
3
1
3
Data access
latency
3
Physical location
irrelevancy
3
Common search
engines
Open interfaces
and standard
protocols
1
3
2
Machine-tomachine service
interfaces
Custom processing
2
2
1
2
1
1
3
3
2
2
3
Data pedigree
2
1
3
Fine-grained cost
control
2
Accessible expert
knowledge
Community
feedback
Rich access to
EOS data
20
1
1
1
1
1
2
PoDAG 25: Data Systems
2
3 – High Correlation
2 – Moderate
Correlation
1 – Some Correlation
Blank – No Correlation
Identified
Evolution status and plans for ECS:
Timeline for ECHO-WIST Transition
•LP DAAC
DAR tool is
in OPS and
decouples
MSS user
profiles
from logins
its DAR
tool logins
•Testing
starts for
End to End
All-Guest
access to
ODL Orders
and Sub
setting using
WIST + V0
Adaptors
•ECHO ready
for testing
SOAP
Orders, Sub
setting and
ASTER OD,
but needs
UAH, ECS &
LP DAAC
components
•End-to-End
ODL AllGuest
access from
WIST ECHO 9 to
Legacy SDPS
is in OPS
•ECHO
continues to
interoperate
with ECS
through the
V0 Adaptors
ECS REL
7.20
EXTEST
05/31/07
21
•Retire JDT &
DAR
components
ECHO 9.0
EXTEST
01/2007
•End-to-end
testing starts
for SOAP/
ECHO 9 AllGuest
access to
Sub setting
and Ordering
at NSIDC
•LP DAAC
(TBD) testing
starts for
SOAP AllGuest
access to
ASTER OD
and Orders
ECS OPS
(TBD)
•End-to-end
testing starts
for LP DAAC
MTMGW to
ECHO
•LP DAAC
interfaces
directly to
DORRAN for
billing and
accounting but
needs EWOC
•OPS continues
ODL support
until ECHO User
Profiles
•End-to-End
SOAP AllGuest
access from
WIST to
“Final
Release” is
in OPS
•ECHO ready
for testing
End-to-end
thread for
Registered
User access
to all threads
but needs
ECS Patch
•End-to-end
testing starts for
Registered User
access from
WIST - ECHO 10
to SDPS
•REL 7.20
LP DAAC
OPS
12/31/06
•JDT and
DAR
Support
•User
Profile Mgt
for DAR
User Logins
to LP-JDT
•UAH ready
for testing
SOAP Sub
setting but
needs ECS
EWOC
•SOAP I/F to
Ordering and
Sub-setting
ECHO
9.0
OPS
04/01/07
JAN2007
APR2007
UAH
EXTEST
05/31/07
•HEW Subsetter: SOAP
I/Fs
MAY2007
LP DAAC
EXTEST
07/2007
•Traffic Cop
I/F in SOAP
•B&A I/F with
DORRAN
ECS 7.20+
EXTEST
07/01/07
ECS OPS
•EWOC
LP DAAC
OPS
08/2007
•Traffic Cop
•B&A I/F with
DORRAN
LP DAAC
EXTEST
08/2007
•EWOC:
ECHO-WSDL
Ordering
Component
•MTMGW to
ECHO
JUL2007
AUG2007
ECHO 10.0
EXTEST
09/2007
•User Profile
Management
Tools &
Services
SEP2007
PoDAG 25: Data Systems
•Final ECS
Evolution
Release
•MTMGW to ECHO
UAH OPS
•SOAP sub
setter
•Target WISTECHO SW
architecture
•ECS-V0
adaptors can
be disabled
•EDG can be
retired
•All searches
and browse
serviced by
ECHO
•All registered
user access
managed by
ECHO
•Obsolete
ECS
components –
V0GWY,
MTMGW, MSS
user profiles
LPDAAC OPS
ECS 7.20+
EXTEST
(TBD)
•Remove User
Profile accesses
from ECS Custom
Code
•Incorporate final
user profile
design into OMS
contact info
OCT2007
ECS 7.20+
OPS
•All legacy profile
accesses removed
ECHO 10.0
OPS
•User Profile
Management
transitions to
ECHO in OPS
DEC2007
ECS REL 7.21
EXTEST
03/21/08
•Retire SDSRV,
V0GW, MTMGW,
MSS User
Profiles
•Retire Sybase
Replication
Tables and
COTS
MAR2008
Evolution status and plans for ECS:
Operations Concept for EWOC (Tentative)
4i. Send email to user
External Subsetter
EMD
4d. Order
4e. Order
4f. Update Order Status
User
OMS
4a. Order
4c. Place order
2. Search
EWOC
WIST
4g. Order Status Update
3. Browse
4h. Order Status Update
4. Order
4b. Register order and request
ECHO
1a. Export metadata
BMGT
1b. Export browse
22
PoDAG 25: Data Systems
MSS
Conceptual Layered Architectural: One Vision for an Evolved
NSIDC Science Information System
User
Communities
External
Systems
Message
Router/Gateway
Interface Services
ftp
http
GISMO/PSQ/WSRD(JAVA)
Search and Retrieve Service
WSDL/SOAP Server
adapters
Custom Visualization
WMS
Custom Search
Reformat Service
Data
Provider
RSS Feed
OWS
NISE/NRTSI Processing
AVHRR Processing
EASE-Grid Processing
SSMI Sea Ice Processing
SI IndexGenerator
WCS
Catalog Service
Inventory (granule/file) Service
Data Operations Metadata Service
Preservation Metadata Service
Subset | Stitching Services
Grid | Regrid Service
Project | Reproject Service
WFS
PerlDesk
gzip, tar
Metrics Reporting System
TRAK
Ingest Services
•Data Transfer
•Validation
•Checksum
•Extract/Generate
Metadata
•Insert to Data Store
•Trigger Subscriptions
(Data, Metadata, Browse)
Archive
Storage
Metadata
Database
Replicate
Working
Storage
Recover
DVD/CD Production
Core Infrastructure Services
Problem Tracking System
SNIPS
AntiVirus System
Intrusion Detection System
23
Sybase RDBMS
Storage Management System
(StorNext/AMASS )
Postgresql/PostGIS
MySQL
LDAP
SNMP
Legato Backup System
PoDAG 25: Data Systems
Apache
Tomcat
Copan
MAID
Vtl
(backup/archive)
Offsite
backup