23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
-1-
Algorithm Preservation versus
Data Preservation
How to Manage Value-Added Processing
in Earth Observation Payload Ground Segments
Martin Boettcher, Werum Software & Systems AG
PV 2005, Edinburgh, 23/11/2005
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
-2-
Overview



Processing scenarios
Processing management functions
Processor interfacing
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
-3-
Systematic Processing – GOME L3 Example
O3
O3
10110010
00010101
10111001
01000100
700
600
500
300
240
Generation of GOME L3 products with global coverage
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
14 or 15
L2
products
per day
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
O3
L3
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L3
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
L2
one L3 product per
day out of 7 days of
L2 input
Use of several days of L2 inputs for a single GOME L3 output
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
-4-
L2
L2
L2
L2
Processing on Request – MODIS L1b Example
Prod.
Request
MODIS
L0
MODIS L1b
Processing System
MODIS
L1b
Precise
Orbit
Attitude
Generation of a MODIS level 1b from level 0 on request
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
-5-
Processing Scenarios


Data-driven scenarios

triggered by new input product (NRT, offline)

timer-based, regular

processing of standard products for subscription handling

processing of customer-defined products for standing requests

systematic quality assurance and sensor monitoring

reprocessing (systematic processing with “large delay”)
Request-driven scenarios

post-processing of existing products on customer request

processing of producible products on customer request

processing of customer-defined future acquisitions (e.g. TerraSAR-X)
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
-6-
Features of Systematic / On-demand Processing
Systematic Processing
Processing On-demand
All high level products are generated in advance
even if only some of them will ever be requested.
Only products that are requested will be produced.
High level products require additional storage space
in the archive.
Only the low level products (L0) require space in the
archive
High level products are offered as existing, they may
be quality checked before they are offered.
High level products are offered as producible, but
generation may fail.
High level products can immediately be delivered.
Processing time delays delivery at time of request.
Standard processing options must exist. The results
of processing are standard reference products.
Individual processing options from user requests
can be supported. The results of processing may be
customized delivery products.
Subsequent processing chains and subscriptions
on the high level product can be supported.
The same product may have to be regenerated if
several customers request the same.
Reprocessing is required whenever better
algorithms or parameters are available.
Better algorithms or parameters can be used as
soon as they are available.
Computing power is required for processing of all
products and for reprocessing.
Computing power is required for the amount of
requests.
Preservation of high level products is required.
Preservation of an operational algorithm is
required beyond end of the mission.
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
-7-
Generic Functions of Processing Systems
request
driven
processing
systematic
data driven
processing
(trigger)
input or
auxiliary products
determination
Processing System
Generic Functions
Specific Functions and Structures
input retrieval
output cataloguing
& archiving
Products
request
queuing and
scheduling
managed
processing
cache
Requests
Workflow
Rule
Workflow
WorkflowRule
Rule
output
dissemination
workflow
control
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
Prim. Processor
QL Generation
Metadata Extraction
…
timer
control
-8-
ingestion
pickup
point
external
data provider
Generic Functions – Uniform Operating
Example Views of Requests,
Products and Processing Steps of a DIMS Processing System
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
-9-
Generic Functions – Interaction with the Infrastructure
Product Library
Input and output
handling
Archive
Inventory
input product retrieval
Processing System
PSM
output cataloguing & archiving
Data
Processor
Metadata
Product Library
Archive
Processor
Processing System
Inventory
PSM
auxiliary product determination (query)
AUX determination
Workflow Rule
Data
Product Library
Archive
Processor
Metadata
Inventory
Trigger
trigger registration
Processing System
PSM
input availability notification (trigger)
Data
Processor
Metadata
Product Library
Archive
Processor
Inventory
Processing System
input determination (timer/query)
PSM
Timer and query
Processor
Data
Processor
Metadata
Product Library
Archive
Inventory
Reprocessing
input determination (query)
PSM
Workflow Rule
Data
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
Processing System
Processor
Metadata
- 10 -
Generic Functions of Processing Systems

to be provided in a management layer

integrated into the ground segment infrastructure
processor integration with adaptable interfaces



support for the different processing scenarios
(data driven, request-driven, …)
support for simple and complex cases
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
- 11 -
Processor Integration – Control Interface
Processing System Management / Distributed Processing Node
Java
Callback
Interface
Java
Processor
Interface
Java
Processor
C/C++/Ftn
Processor
Interface
Shared Library
Processor
Processor
Thread
PSM
C/C++/Ftn
Callback
Interface
Executable
Processor
Interface
Executable/Shell Script
Processor
Adapter
PSM
Executable
Callback
Interface
Executable
call
call
execute
callback
callback
commands
commands
communication
channels and filters
(stdin, stdout, signals)
termination
return
return
In-process execution
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
child process generation
- 12 -
Processor Integration – Data Interface
Cache
Directories
input product
primary &
metadata
input product
primary &
metadata
reads
reads
reads/writes
working directory
output product
Processor
writes
primary
quicklook
metadata
...
Processor’s file interface to products
(example with two inputs and one output)
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
- 13 -
Guidelines for Writing Bad Processors







Write intermediate files into local directory of the
processor software
Rely on correct environment and inputs,
this makes error handling superfluous
Run for hours without possible breakpoints,
crash after restart on partial results
Generate either lots of debug output in operational mode,
or run without any progress feedback
Implement management functions into the processor
to ensure perfect embedding into the infrastructure
Use specific features of the hardware, the operating
system, or the data source
Require some files from the implementer’s home
directory
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
- 14 -
Conclusion




There are various processing scenarios, among
them systematic and on-request.
A generic processing management layer can relieve
processors from management functions.
Operational processors have to provide a control
and a data interface.
Certain features of processor implementations
influence maintainability in an operational multimission environment.
23/11/2005 © Werum Software & Systems AG
PV 2005 Algorithm Preservation versus Data Preservation
- 15 -