Ocean Observatories Initiative OOI CI Overview and Status Matthew Arrott DMAC-ST Washington DC, Jan 18-19, 2012 OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 1 Agenda • Review of the OOI CI Construction Objectives • Introduction of Release 2 Functional Objectives • Computational Infrastructure Objectives • External Observatory Integration • What will OOI-CI mean to me OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 2 OOI CI Construction Objectives OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 3 OOI Project Scope Building/deploying science infrastructure for understanding a changing water planet • Multi Dimensional Engineering Concern • Social • Organizational • Functional • Procedural • Technical Cyberinfrastructure fulfills the integrative mechanism across these dimensions OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 4 The Hubble Telescope for the Oceans OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 5 The Hubble Telescope for the Oceans OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 6 Core Science & Education Capabilities • Interactive Ocean Observing (R3 - R4) • Interactive Ocean Modeling & Data Assimilation (R3 - R4) • Discipline-Driven Semantic Organization of Data (R3 - R4) • Automated Data Product Generation (R1 - R3) • Interactive Instrument Network (R1 - R3) • Integrated Observatory Management (R1 - R2) • User-Driven Integration of Resource (R1 - R3) Observatory Activity Model 7 OOI CI Release 2 Kickoff May 2, 2011 OOI Integrated Observatory Product •The fully operational research observatory will meet the following •Goals: • • • • • • • • • • • • Continuous observations at time scales of seconds to decades Spatial measurements from millimeters to kilometers Sustained operation during storms and other severe conditions Real-time or near-real-time data as appropriate Two-way transmission of data and remote instrument control Power delivery to sensors between the sea surface and the seafloor Standard plug-n-play sensor interface protocol Autonomous underwater vehicle dock for data download/battery recharge Access to deployment and maintenance vehicles that satisfy the needs of specific observatories Facilities for instrument maintenance and calibration A management system that makes data publicly available An effective education and outreach program OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 8 Extending the Traditional Data Access Model OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 9 Multi-Scale Research Specific Ocean Laboratories 10 Release Schedule Inception Phase Elaboration Phase Construction Phase Transition Phase 11 OOI Review Year 2 May 16-20, 2011 Release Schedule by Subsystem Inception Phase Elaboration Phase Construction Phase Transition Phase 12 OOI Review Year 2 May 16-20, 2011 CI Components Developed for R1 • Python Capability Container • Java Capability Container Access Library • Web UI Platform • Exchange Messaging System • Distributed State Infrastructure • Resource Registry Framework • Science Data Persistence and Transport Format • Data Publish-Subscribe Framework • Event Notification Framework • Instrument Agent Framework • Elastic Processing Unit • Virtualized Cloud Management Tools 13 OOI Review Year 2 May 16-20, 2011 Capabilities of the Managed Instrument Network • Operate Marine Observatories • Operate Platforms and Instruments • Manage Instrument Lifecycle • Test and Troubleshoot Instruments • Acquire Data and Generate Data Products • Search Data • Visualize Data • Manage the Integrated Observatory Network 14 OOI Review Year 2 May 16-20, 2011 R1 Integrated Observatory Network Web UI https://confluence.oceanobservatories.org/display/CIPUB/OOI+Review+2011+Demo s 15 OOI Review Year 2 May 16-20, 2011 Release 2 Functional Objectives OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 16 Observatory Operations & Administration OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 17 User Experience Incremental Development Plan • CI Release-1: Data Distribution Network • Provide framework for discovery and interaction based on data-intensive users’ needs • CI Release-2: Managed Instrument Network Science Experts • Provide support for operations and maintenance of Resource Providers marine networks based on RSN and CGSN needs General Science Users • CI Release-3: On Demand Measurement Processing Scientific Collaboration • Provide dynamic analysis and visualization tools based on science communities’ user needs • CI Release-4: Interactive Ocean Observatory • Provide mission control and collaboration support based on science communities’ needs 18 OOI Review Year 2 May 16-20, 2011 R2 Product Description: List of Use Cases UC.R2.01 Acquire External Data SourceUC.R2.02 Derive Data ProductUC.R2.03 Produce Real-Time QC DataUC.R2.04 Browse to Get Data ProductUC.R2.05 Register and Connect InstrumentUC.R2.06 Command InstrumentsUC.R2.07 Direct Instrument Access IIUC.R2.08 Manage Instrument LifecycleUC.R2.09 Activate Instrument DriverUC.R2.10 Manage Marine PlatformUC.R2.11 Operate Marine ObservatoryUC.R2.12 Deploy Agents On Remote PlatformUC.R2.13 Acquire Data From InstrumentUC.R2.14 Monitor an InstrumentUC.R2.15 Qualify Instrument InterfaceUC.R2.16 Install Instrument Automatically UC.R2.20 Annotate ResourcesUC.R2.21 Transform Data in WorkflowUC.R2.22 Version ResourceUC.R2.23 Ingest Dataset SupplementUC.R2.24 Search for ResourceUC.R2.25 Advanced Resource SearchUC.R2.26 Navigate Resources and MetadataUC.R2.27 Manage Replicated ArchiveUC.R2.28 Manage Resource MetadataUC.R2.29 Integrate External Data Source 19 UC.R2.30 Define InteractionUC.R2.31 Define New ServiceUC.R2.32 Conduct NegotiationUC.R2.33 Enroll in an OrgUC.R2.34 Share an Org ResourceUC.R2.35 Share Affiliated Orgs' ResourcesUC.R2.36 Create an OrgUC.R2.37 Control Service InteractionsUC.R2.38 Define Resource Life CycleUC.R2.39 Manage ION UsersUC.R2.40 Monitor ION ResourcesUC.R2.41 Recover Failed ProcessUC.R2.42 Define Resource PolicyUC.R2.43 Operate Message Brokers DM COI CEI S&A A&S UX UC.R2.17 Define Visualization MethodUC.R2.18 Visualize Data ProductUC.R2.19 Produce Matlab Visualization OOI Review Year 2 May 16-20, 2011 UC.R2.53 View Modeler-Submitted ProductsUC.R2.54 Access NEPTUNE CA Data Ops EOI UC.R2.55 Manage Help TicketUC.R2.56 Monitor ION NetworkUC.R2.57 Configure Start Page UC.R2.44 Put Services Anywhere EasilyUC.R2.45 Replicate Activated ServiceUC.R2.46 Operate Integrated SystemUC.R2.47 Deploy Versioned User ProcessUC.R2.48 Schedule User-Defined ProcessUC.R2.49 Deploy Distributed ProcessesUC.R2.50 Define Scaling PolicyUC.R2.51 Define Execution EngineUC.R2.52 Manage ION Processes UC.R2.58 Display Arbitrary ResourceUC.R2.59 Generate New Screen Managed Instrument Network OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 20 Sensor Set #1 Instrument Agents Class # Family Instrument Class Long Name ADCPS 16 Water Velocity Velocity_profile_600m CTDGV 24 Conductivity, Temperature, Depth CTD_glider CTDMO 135 Conductivity, Temperature, Depth CTD_mooring CTDPF 30 Conductivity, Temperature, Depth CTD_profiler DOSTA 61 Dissolved Oxygen oxygen_dissolved_stable FDCHP 7 Direct Covariance Flux flux_direct_cov_HP FLORD 35 Fluorometer Fluorometer_two_wavelength METBK 18 Meteorology Meteorology_bulk NUTNR 22 Nitrate nutrient_Nitrate OPTAA 35 Optical Properties attenuation_absorption_optical PCO2A 10 Partial Pressure CO2 pCO2_air-sea PCO2W 25 Partial Pressure CO2 pCO2_water PHSEN 33 Partial Pressure CO2 pH_stable PREST 3 Seafloor Pressure pressure_SF_tidal SPKIR 22 Optical Properties spectral_irradiance VEL3D 32 Water Velocity Velocity_point_3D_turb VELPT 21 Water Velocity Velocity_point WAVSS 10 Surface Wave Spectra wave_spectra_surface ZPLSG 4 Zooplankton Sonar 21 OOI Review Year 2 May 16-20, 2011 plankton_ZP_sonar_global Sensor Set #2 Instrument Agents Class # BOTPT Family 3 Seafloor Pressure Instrument Class Long Name pressure_bottom_tilt CAMDS 12 Cameras x CAMHD 1 Cameras x camera_digital_video_HD HYDLF 5 Hydrophone Hydrophone_LF_passive MASSP 2 Mass Spectrometer mass_spectrometer OBSBB 3 Ocean Bottom Seismometer Seismometer_BB_triaxial_accel OBSBK 2 Ocean Bottom Seismometer Seismometer_BB_triaxial_keck OBSSP 8 Ocean Bottom Seismometer Seismometer_shortperiod PPSDN 1 Seafloor Fluid and DNA DNA_particulate RASFL 1 Seafloor Fluid and DNA watersample_chem_trace_H2S_pH THSPH 1 Seafloor Properties Temp_H2_H2S_pH TMPSF 1 Seafloor Properties Temperature_seafloor TRHPH 1 Seafloor Properties Temp_resist 22 OOI Review Year 2 May 16-20, 2011 camera_digital_still_strobe Sensor Set #3 Instrument Agents Class # Family Instrument Class Long Name ADCPA 15 Water Velocity velocity_profile_mobile_asset ADCPT 17 Water Velocity Velocity_profile_300m CTDAV 3 Conductivity, Temperature, Depth CTDBP 18 Conductivity, Temperature, Depth CTD_bottom_pumped DOFST 22 Dissolved Oxygen oxygen_dissolved_fastresp FDCLP FLORT 3 Direct Covariance Flux 42 Fluorometer CTD_AUV flux_direct_cov_LP Fluorometer_three_wavelength HPIES 2 Inverted Echo Sounder IES_pressure_velocity HYDBB 6 Hydrophone Hydrophone_BB_passive NUTR4 3 Nutrients nutrient_four_channel OSMOI 2 Seafloor Properties watersample_chem_trace OTISF 1 Seafloor Properties flow_benthic PARAD 30 Optical Properties PRESF 9 Seafloor Pressure pressure_SF VADCP 2 Water Velocity Velocity_profile_50m_turb ZPLSC 8 Zooplankton Sonar plankton_ZP_sonar_coastal 23 OOI Review Year 2 May 16-20, 2011 PAR CI Architecture for Instrument Integration 24 OOI Review Year 2 May 16-20, 2011 Data Processing and Product Generation 25 OOI's Sensor Life Cycle OOI Final System Design Core Sensor List Core Sensor Specs Sensor Working Group Core Sensor RFPs Common Sensor Selections COL COL RSN RSN CGSN Deploy at Sea RSN CI System Integration Test CI Integration Test CI CI COL 26 RSN CGSN CGSN CI CI CGSN Procure Common Sensors OOI Review Year 2 May 16-20, 2011 Test w/Instrumen t Test Kit Encode Algorithms Develop QC/Xform Algorithms RSN CI Integrate w/ Sensor Agent CI CGSN Develop Sensor Drivers Instrument Development Kit • Scheduled for Release-2 • Includes: • Logical Test Facility Workbench (dry testing) • Marine Specific System Test Facility (wet testing) • Configurations for RSN and CGSN observatories • Access and Management Portals for Interactive access 27 OOI Review Year 2 May 16-20, 2011 Computational Infrastructure Objectives OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 28 National Network Deployment (Year 3) 29 OOI Review Year 2 May 16-20, 2011 CyberPoPs and Network Infrastructure • Acquisition Point CyberPoPs: • Portland, OR • Woods Hole, MA • Distribution Point CyberPoPs: • Seattle, WA • Chicago, IL • San Diego, CA • McLean, VA (optional) • Engineering Center: • San Diego, CA • Network Infrastructure • Dedicated 10GE loop San Diego, Portland, Seattle, Chicago; branch to McLean (optional) • Dedicated 1GE connection to Woods Hole 30 OOI Review Year 2 May 16-20, 2011 San Diego Engineering Center • Purchased and deployed San Diego CyberPoP and network equipment (see rack drawing) • Located in UCSD’s Atkinson Hall server room 1101 (secure and protected) • In use for • Release-1 production and QA • Management tools (Confluence, Jira, etc) 31 OOI Review Year 2 May 16-20, 2011 San Diego CyberPoP and Network Equipment 32 OOI Review Year 2 May 16-20, 2011 Acquisition Point CyberPoP Deployments (Y3) • Sites • Portland • Woods Hole • Function • Data acquisition from marine observatories • Real-time data processing and OOI data product generation • Data preservation • High availability 33 OOI Review Year 2 May 16-20, 2011 CyberPoP Physical Layout (Year 3 Deployments) • Sites • Seattle • Chicago • Function • Content distribution (web servers, data servers, messaging) • Interconnects with major national and international network providers (layer 2 peering) • Links to compute clouds 34 OOI Review Year 2 May 16-20, 2011 Network Peering Integration Seattle OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 Chicago 35 Multi-Tier Messaging Federation OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 36 Multi-Site Service Network Deployment 37 External Observatory Integration OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 38 IOOS Integration Model OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 39 External Observatory Integration Progress • Identification of datasets and data sources needed to support early adopter group (ESPRESSO/Rutgers, HiOOS/UHawaii) numerical model workflows • Classification of datasets by type and representation • Development of Dataset Agents for automatic data and metadata ingestion • Translation of data/metadata into the Common Science Data Format • Streaming of data packets and metadata update notifications 40 OOI Review Year 2 May 16-20, 2011 External Observatory Integration • Exemplar: MARACOOS (Rutgers University) • Experimentally substitute part of the scientists’ data assimilation workflow preceding numerical model execution 41 OOI Review Year 2 May 16-20, 2011 External Observatory Integration: Master Dataset List • Datasets • NAVY : NRLSSC (DAP) : 1 • NOAA : NDBC SOS (HTML) : 34 • NOAA : NDBC SDF (DAP) : 1 • NOAA : PFEG (DAP) : 1 • Rutgers : tashtego (DAP) : 2 • UCSD : HFRNET (DAP) : 1 • UH : SOEST (DAP) : 1 • USGS : WaterService (HTML) : 26 42 OOI Review Year 2 May 16-20, 2011 “What will OOI-CI mean to me” OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 43 Regional Data Provider & Manager • Effective mechanism to reliably at scale distribute your data • to your user community and data management/preservation facility • Independently publish and receive data • in a wide variety of syntactic formats and community vocabularies • Publish and receive as near-real-time data streams and as historical data sets • Effective mechanism to reliably at scale aggregate and disseminate data from your publishers and to your end user communities 44 OOI CI Release 2 Kickoff May 2, 2011 Participating Agency • Participate as providers, managers and consumers • Anticipated some requirements for publishing on the OOI as a separate Agency will require coordination with NSF • Requirements are being addressed as a part of the IOOS OOI Integration effort 45 OOI CI Release 2 Kickoff May 2, 2011 How does one “join” – approval / certification? • Open data policy network for consumers of data • Anticipate publishers required to register and certified • Data policies and rights management being established • Open data policies and content limitations to science and education data content • Publishing requires acceptance of some form of open use policy • Consuming requires some form of “as is”; “at your on risk”; “with attribution of source if republished” policies 46 OOI CI Release 2 Kickoff May 2, 2011 Relationship with IOOS, NEPTUNE and others observatories • External Observatory Integration subsystem within the OOI • Specifically focused on providing bi-directional integration interfaces to IOOS, NEPTUNE-Canada and WMO (aka GTS) • Integration continues on into the Operations and Maintenance phase of the OOI as a user/observatory “on boarding” capability • Provide the training and support to communities that wish to integrate with OOI 47 OOI CI Release 2 Kickoff May 2, 2011 OOI define standards for them to “plug in”?, e.g SensorML • OOI provides a wide variety of syntactic formats and community vocabularies by which to interface with the OOI • OOI and the user community will be able to extend these interfaces • Capability facilitated by architectural choice of using controlled set of canonical data models into and out of which all data are transformed • New transformations are added, “plug ins”, to the network for a new format and/or vocabulary to one of the controlled data models 48 OOI CI Release 2 Kickoff May 2, 2011 Will a region be able to host an instance of the OOI “stack”? • The OOI “Integrated Observatory Network” is a federation architecture • Comprising participants from multiple domains of authority operating from their local domain of authority • The OOI “stack” exists at a couple of levels: • It is a messaging protocol • It is a component software implementation that implements the messaging protocol and process management containers (think modern web server) that publish and consume services • “Capability Containers”, can be and are written in multiple languages • achieve their interoperability through use of the common OOI messaging protocol • OOI message protocol and any of the OOI supplied Capability Containers will be provided under one of the standard open source licenses 49 OOI CI Release 2 Kickoff May 2, 2011 Thanks! OOI CI R2 Life Cycle Objectives Review Aug 30 - Sep 1 2011 50