MONITORING WEATHER
AND CLIMATE FROM SPACE
Challenges in the provision of large data
sets to multiple user communities
14-16 July 2014, Boulder CO
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
1
Mission Statement
We are the European Organisation for the Exploitation
of Meteorological Satellites (EUMETSAT).
EUMETSAT is an intergovernmental organisation and was
founded in 1986. Our purpose is to supply weather and climaterelated satellite data, images and products
• 24 hours a day,
• 365 days a year
to the National Meteorological Services of our Member and
Cooperating States in Europe, and other users worldwide.
www.eumetsat.int
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
2
EUMETSAT Headquarters (Darmstadt, Germany)
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
3
EUMETSAT Data Services - High Level Overview
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
4
EUMETSAT Data Services - Data Flows
EUMETSAT GROUND INFRASTRUCTURE
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
5
EUMETSAT Earth Observation Portal
 Single Access Point
 One stop shop
 Data discovery
 User registration (>10.000)
 Subscription for data
services:

Near-real time data

Online Data Access

Long Term Archive
 Additional

Image Gallery (WMS/ WCS)

User Notifications

Operational Real-Time
Services Status Indicator
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
6
EUMETCast Dissemination
Near Real Time data delivery to users in 3 continents via Satellite
4
3
500+
201-500
51-200
11-50
2-10
1 User
0 User
Users (2/2014)
3,280 users worldwide,
4168 stations
5 new users per month
EUMETCast Americas
SES 6, C-BAND
EUMETCast Africa
EUTELSAT 5 WEST A, C-BAND
EUMETCast Europe
EUTELSAT 9 A, KU-BAND
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
7
EUMETCast Dissemination:
Integrating terrestrial dissemination
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
8
MULTICAST ACCROSS GÉANT and NREN
INFRASTRUCTURE
Exploring the multicast usage on
GÉANT infrastructure in Europe
and their world wide connectivity;
 2011 Proof of concept on terrestrial
multicast on GÉANT test bed
 2012 3 month feasibility study on
terrestrial multicast via GÉANT
 2013 Multicast test to KMA Korea
 2014 Multicast test to NOAA
(Boulder)
 2014 Start of 2 year prototype
emulating operational scenarios of
multicast across Europe and our
overseas partners using GÉANT,
NREN, I2 and Teint infrastructure.
EUMETCast
Platform
(MMDS)
EUMETCast
User Stations
Multicast
GEANT/
NRENs
Multicast
...
Multicast
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
9
Reference
Stations
EUMETCast Terrestrial real-time monitoring
-
Multiple Multicast channels
SSM mode
20-30Mbit/s
FEC ¾
Closed loop monitoring based on successful file receipt
Packet/channel based ack/nack retransmission possible but not yet enabled
Acknowledgements: N-WAVE&NOAA Alex Hsia, David Hartzell
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
10
EUMETCast Dissemination:
Low cost, off the shelf Receiving Station
DVB Router
Laptop/ PC
Antenna
DVB PCI Card
Software
EKU
DVB USB
Typical total cost of a station < 2000 Euros
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
11
EUMETSAT Long Term Archive
 Archived data starting 1981
 All EUMETSAT Satellite data
including reprocessed data
 > 1 Petabytes in the archive
for MTP, MSG, Metop
 Retrieval/Ingestion: 10 to 1
 Long-Term Data Preservation
in practice since 1995;
 Disaster resilient - 3rd Copy
 Access via EUMETSAT EO
Portal
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
12
What we are working on…
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
13
Outlook – Enhance Future abilities in terms of
•
•
•
•
•
Scalability
Flexibility,
Responsiveness to emerging needs,
Resource efficiency
While aiming for:
• Better sharing of resources,
• Avoidance of data islands,
• Reducing costs
• Fit the organisational structure
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
14
Many aspects to consider => Challenge
Viewing Angle
15
EUM/TSS/VWG/14/762641, 7 July 2014
Outlook - Product delivery and management
• Generation of user tailored products
• Efficient for write once, read many times
• Suitable products for user needs and
dissemination infrastructure (bandwidth)
• Avoid/ minimise processing where possible
• Efficient for slicing
• Usage of Format evolutions
• Secure the raw data (golden copy)
• Initially smaller products which increase over
time
• Reprocessing to re-build formats
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
16
Outlook - Bringing data to Users
• Adaptive and flexible services to deliver data to
users/ communities (e.g. EUMETCast)
• Community driven formats optimised for
dissemination mean
• Serving large user groups from different
communities
• Managing data policies in accordance with data
provider requirements via a data policy and
legal framework support by the related
technical mechanisms
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
17
Outlook - Bringing Users to data
 Standardisation activities in relation to Discovery,
Access and Retrieval (DAR) of data
 Strong focus on interoperability and ease of user
experience
 One stop shop experience for discovering services and data
 Facilitating the Interoperability of portals by exploiting standards
(OGC, INSPIRE)
 Enhance Long Term Archive, Online Data Access
 Outreach activities
 Exploiting capabilities to execute user algorithms on data
 Post-Processing, Hosted Processing
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
18
Outlook - Information to Users
• Delivering Information to users rather
than data
• Processing where the data is located
• Service Interfaces instead Facility
interfaces
• Application of (private) cloud principles
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
19
Thank you for your attention
Improving
Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int
20