Netlobs
Manipulating Gridded Data in a
Relational World
Neil Stamps
Technical Architect
Agenda
Introduction to Lost Wax
The problem framed
Oracle 10g capabilities
NetLobs – NetCDF meets database
Introduction to Lost Wax
Server-side Systems
Multi-agent Systems
Continual R&D
Roles based business analysis
Agent framework
Products (SPL)
Innovation projects
J2EE / MS.NET
SOA and Web Services
Legacy integration / wrapping
GIS Mapping Solutions
Advanced
Software
Engineering
Mobile
Distributed computing
Java PDAs / phones
The problem framed
DTI DEWS research project partners:
Met Office, Reading University, BADC, BMT, IBM
SQL
Forecast Data
GADS
Interne
t
Web
Service
Provides web services to multiple domains
GADS provides marine services
Oracle target platform
Oracle – Blob support
Oracle 10g deployment platform
Large object support
Blob – max size (4GB –1)*block
e.g. 32k block = 128TB max
Clob, nClob – max size as per Blob
Extension support
Java, C extensions
Java stored procedures
Custom data types (cartridges)
Remote symbolic debugging (JDeveloper)
Oracle - Custom data types
Provide encapsulation of attributes and
methods
Introduce OO capabilities into relational world
Allow unstructured data to be queried
Extensions to indexes allow efficient queries
Nested tables provide collection capability
NetLobs – NetCDF ‘SmartLobs’
Provides NetCDF file capability to Oracle
Encapsulates data and meta-data in single type
Physical implementation agnostic
Automatic extraction and storage of meta-data
Interrogate meta-data without blob enquiry
Extraction over single or multiple Netlobs
NetLobs – NetCDF ‘SmartLobs’
NetCDF 2.2 open source Java libraries
NetLob wraps Oracle for NetCDF Files
Extraction interfaces based upon current
GADS requirements:
Subset
Reduction
Concatenation
Higher level interfaces to be layered over
basic functionality
Cartridge invocation
PL/SQL interface maps to Java call (or C)
Oracle instantiates NetLob object
Object implements SQLData interface
Blob pointer passed to Java, Random access
provided via ‘internal’ JDBC
NetLobs – Data ingestion
Upload using Oracle SQL*Loader
Upload in two–phase method
Validation at Netlob creation
System optimisation based upon once-only
performance hit at extraction of meta-data
Meta-data ‘chunks’ will facilitate query by value
NetLobs – Performance
Reference GADS system provides predictable,
linear extraction performance
NetLob cartridge aims to achieve similar
performance characteristic over large data
extractions
Optimisation tailored to larger extractions
Moving forward
Storage and retrieval of rotated data
Pluggable interpolation framework
Offloading processing to GRID
Enhanced meta-data to meet community needs
Query by-value enhancements
Any Questions?
Neil.Stamps@Lostwax.com
www.LostWax.com