PDS Lessons Learned
• The UVIS team should have been allowed to
define PDS data objects.
• Validation of PDS data products should have
been done by an independent agent.
• Derived data (i.e. non-telemetry data) is difficult
to validate and should be kept to a minimum.
• Coordination between PDS nodes and data
providers should be managed by a third party.
• Sophisticated tools for searching PDS are
required.
• PDS should provide mission independent data
product readers.
Data Model
• The choice of PDS QUBEs and Time Series
instead of PDS tables has lead to
– irrelevant data
• ~70% of delivered data is CORE_NULL
– complex validation strategy
• because data was delivered as PDS QUBES, not DAPS
database tables, validation by science team usage was
impossible.
• In order to do validation, a PDS reader (analogous to
feuv_reader, etc.) was written. Intermediate cross-check
code was used to compare PDS objects with NetCDF objects
in IDL.
• PDS reader is independent of NetCDF readers and required
repeated maintenance, e.g. SCTimeFine bug,
“COUNTS/BIN” bug, data volume bug, etc.
• Procedure has been successful in validating data (counts).
Validation
• Validation should be executed by an
independent agent
– science team feedback has been very helpful in
regards to DAPS. For example, SCClock bug was
detected by Alain, start time strings by Miodrag,
calibration coefficient error by Greg Holesclaw, HSP
integration time by Josh.
– Occasional input from PDS users e.g., data volume
assumptions by R. Yelle, Space Science Reviews
citation by ???
– Occasional input from PDS node has also identified
problems. For example, PDS “COUNTS/BIN” bug,
HI_VOLTAGE_POWER_SUPPLY (ON/OFF instead of
integer), calibration file pointer problems (missing
(“_CAL_N”).
Non-Required, Derived Data
• Geometry data is a snapshot taken at the first
integration which is not useful for data analysis.
Furthermore, it is unverifiable.
• Redundant information such as string
representations of the start time provided by the
spacecraft clock are an invitation to error
(consider V1 and V2 of convert_sclk.pro).
• Arcane, unchecked syntax requirements such as
“COUNTS/BIN” instead of COUNTS/BIN or
carriage return/line feed line terminators lead to
unjustified amounts of work.
Coordination of Deliveries
• Deliveries submitted, spot checked by
PDS node, errors fixed and data
resubmitted.
– node representative and data producer tend
to avoid conflict causing problems such as
delays in code release.
– Third party such as SAWG leader could
provide more coordination.
Searching
• INDEX.TAB is the only mechanism for
finding data.
– SQL queries are only means to select data.
– Clients need to completely understand
instrument in order to obtain types of
observations. For example, integration times
greater than ??? indicate high signal to noise
observations.