Comments on version 3.0 toward version 3.1, from May 2014
Thierry, 26/05/2014 13:45
Please find the current draft version of Argo user’s manual version 3.1 on:
http://www.argodatamgt.org/Documentation/Draft-documents
This document was updated from the current version 3.03, the decisions from Liverpool Argo data
management meeting and a series of emails listed in “comments on version 3.0 toward version 3.1”.
Some of us already started to produce 3.1 sample files.
Ann, Sylvie, we now need a green light to accept this files on Argo GDACs.
Annie, 27/05/2014 00:12
Hi Thierry. Thank you for updating this increasingly complex document.
I have 3 comments for this draft.
1. On p64, I think the entire section 2.6.1 (Pressure axis management in core-Argo and b-Argo files)
can be improved. This is the most important section for the coordination between the core- files and
the b- files, and therefore needs to be absolute clear. I am attaching here my attempt to rewrite this
section, with the title changed to "Management of vertical pressure levels in core-Argo and b-Argo
profile files", and have included a hypothetical example. Please take a look at the attached doc.
I hope it helps.
TC: thank you Annie, you text is clear and does not conflict with the existing text. The example you
provide is simple and useful. So I replace the existing chapter §2.6.1 with your text.
2. Again on p64, I suggest you change the title for section 2.6.2 to "Management of multidimensional spectral parameters", or something similar.
The current title "Management of array values" is not very apt.
TC: ok for a more explicit title: change of title “Management of multi-dimensional parameters”. I
preferred to remove the term “spectral” from your title suggestion, a multi-dimensional parameter
may not be spectral.
3. Please update Reference Table 11 to include two new rtqc tests for near-surface data. Their
descriptions have already been recorded in the qc manual. They now need to be included in Table 11
and be given unique binary IDs. They are:
Test 21. Near-surface unpumped CTD salinity test.
Test 22. Near-surface mixed air/water test.
TC: yes, these tests were missing. They are now added in reference table 11.
John, 31/05/2014 02:21
Hi Thierry, A typo in manual for JULD_DEEP_PARK_START which has the long_name="Deep Descent
end date of the cycle". This is an exact copy of the JULD_DEEP_DESCENT_END. Thus the
JULD_DEEP_PARK_START should be different.
Comments on version 3.0 toward version 3.1, April-May 2014
Thierry, 18/04/2014 19:04
Dear Esmee and All,
Jean-Philippe and I copied the update of metadata section in the attached version of the “Argo user’s
manual”.
We think that this version contains a lot of changes compared to the existing official version 3.0.3
from August 28th 2013.
Instead of using an intermediate version “3.0.4”, we should use a major version “3.1” for this manual.
To differentiate the existing Argo 3.0 NetCDF files and the files built from this new manual, we should
also probably use “3.1” for profile, trajectory and metadata files.
This is not an obligation for the format checker; it can combine user manual version (3.1) and file
format (3.0), but I think it would be easier to adopt “3.1” for file format versions.
The word version of the manual is available to all from :
http://www.argodatamgt.org/Documentation/Draft-documents
Ann, Claudia, can you look at page 75 §3.3 “Reference table 3: parameter code table”
The question to include or not DOXY as core-Argo parameter is still open.
Maybe shall we be able next week to propose this manual as the official version?
Mike, 21/04/2014 20:27
Hi Thierry:
Please change the ftp://usgodae1.fnmoc.navy.mil/pub/outgoing/argo to
ftp://usgodae.org/pub/outgoing/argo in the manual Section 4.
TC : done
Ann, 22/04/2014 04:35
I totally agree that all of these changes (and their implementation) require a new version number
and 3.1 makes sense.
Regarding DOXY as a core variable, it doesn't really make sense to split it from the other biovariables. Having said that, I don’t know if Claudia will be able to split her files and that is the big
question. However – she can still provide the older version v3.0 files (which can include DOXY) if she
wishes until she can get the new code broken up into B and C files. The time frame, however, should
be limited.
The problem with Claudia's solution of providing DOXY in the core files without the raw data for the
parameter is that it really messes up anyone who needs to do DMQC for oxygen (or any other bio
variable). I think if she's providing DOXY in the version 3.0 files, then she needs to include the raw
parameters, though that is very unsatisfactory. I realise she's not funded to do bio data but if she's
funded to do oxygen, this is just another aspect of that data management (and will set her up for
other variables down the track perhaps).
I would also suggest even if she has other bio variables she should provide only what she can. It really
is a simple measure to split them into a 'duplicate' file with just the DOXY variables and raw pressure.
Where it gets REALLY complicated is when you try to add FLBB data, nitrate data, EM data… Then
it's a nightmare and DOES require significant programming changes so I've even sort of ignored them
for now…
I'll take a closer look at thereat of the manual but since I've just finished coding profile and metadata
files, I don't expect many issues.
TC : no immediate impact on the manual
John, 25/04/2014 21:42
I have two comments on the temporal resolution attributes added and not added in manual 3.1.
First, Why not add JULD and JULD_LOCATION resolution attributes in the profile file, if we are
requiring it in the traj file?
TC: OK to be homogeneous between prof and traj ont hat topic, added in the manual
§2.2.4 : add “JULD:resolution = X;” and “JULD_LOCATION:resolution = X;”
Second, In the traj file the JULD and JULD_ADJUSTED variables are likely to have variable resolution
depending on the measurement code.
At least I know that is the case with the SOLOII. Thus I'm assuming the same procedure is followed
whereby no resolution attribute is required under the variable, but then a global comment is added.
This should be added to the manual, so that users know it is not only PRES. I also think that TEMP
and PSAL and all other variables in the traj file could vary well have different resolutions, so phrasing
allowing multiple variables to have this handling should be added.
TC: this information should be reported in the global attribute “comment_on_resolution”
described on §2.3.1 page 30.
Claudia, 28/04/2014 21:10
we will not have the ability to distribute raw oxygen data in format 3.x.
TC: Ann answered below
Ann, 29/04/2014 02:01? Answer to Claudia
Actually, I just checked version 3.01 of the manual and there isn't a mention of not providing raw
parameters for derived data so it would be acceptable (and that's what we're doing right now in
version 3.03 files and no one has screamed about it yet). That restriction only came in with version
3.1 I would guess. So providing raw values is perfectly fine. If you want an example of one of our
version 3.03 files, just let me know. I also have examples of version 3.1 C and B files…
TC: no impact on manual
Thierry, 30/04/2014 12:35
I agree with you, this parameter should be renamed « PHASE_DELAY_DOXY” (16 letters).
I think that the unit should be “microsecond” instead of “micro seconds” (the singular is used in Argo
units, no plural).
Its valid_min is probably 0. Do we have a valid_max value ?
Ann : This looks sensible to me, Thierry. I've never been comfortable with the phase_doxy name
since it's not obvious how it differs from the other phases. This makes it very clear.
TC : done
Jean-Philippe, 30/04/2014 14:12
Exactly
BPHASE_DOXY and DPHASE_DOXY are reported by Aanderaa 3830 optodes
TPHASE_DOXY and C1/C2PHASE_DOXY are reported by Aanderaa 4330 and 4330F optodes
PHASE_DOXY is reported by SBE63
Jean-Philippe, 30/04/2014 16:11
J'ai recopié le contenu de la nouvelle ref table 3 pour le décodeur Remocean (cf. fichier joint si tu en
as besoin pour les gliders).
Voici les remarques notées que j'ai corrigées pour le décodeur et qu'il faudrait reprendre (ou pas)
dans le manuel et surtout dans le fichier argo-parameters-list-core-and-b.xlsx
- AUM 3.1: colonne long_name: pour homogènéiser remplacer "by the oxygen sensor" par "by
oxygen sensor"
- AUM 3.1: colonne long_name: remplacer "x nanometers" par "<x> nanometers" (pour faire
comprendre que <x> sera remplacé dans le décodeur par la meta-donnée ad hoc (comme pour les
<SENSOR> des labels config BIO)
- AUM 3.1: colonne long_name: remplacer double espace par un seul (pour BETA_BACKSCATTERING,
UP_RADIANCE et DOWN_PAR)
- AUM 3.1: "ABSORBANCE_COR_NITRATE" au lieu de "ABSORBANCE _COR_NITRATE" (un espace dans
le nom)
- AUM 3.1: retour charriot dans long name de MOLAR_DOXY
2 autres remarques sur le manuel 3.1:
- remplacer partout DOWNWELLING_IRRADIANCE_RAW par RAW_DOWNWELLING_IRRADIANCE qui
est le nom dans la liste Excel
- remplacer DOWNWELLING_PAR par DOWN_PAR (incohérence entre liste Excel et manuel, je pense
que l'Excel fait foi car c'est là dessus que nous avions travaillé en dernier avec Catherine) => est-ce la
seule incohérence entre la liste Excel et le manuel?
TC : J’ai remplacé DOWNWELLING_IRRADIANCE_RAW par RAW_DOWNWELLING_IRRADIANCE.
Par contre, DOWNWELLING_PAR est la version longue de DOWN_PAR
Claudia, 30/04/2014 18 :23
Idea:
is it possible to add to the header or foter something that shows if one is
in the meta, traj, prof or tech section of the manual?
TC: sorry, I did not find a way to add a chapter name in the footer. Does anyone know if that is
possible? The PDF version has signs listing the chapters that are displayed on the left of the
document.
The following is following the links in the update section:
=============================================================================
§1.2 : use DOI for data and document citation
TC: I do not see any problem there (?)
=============================================================================
§1.6 : update cycle definition
A cycle is defined as a series of actions made by a float and includes either a descending profile or an
ascending profile (or, rarely, both); it may also include immersion drift or surface drift.
--> A cycle is defined as a series of actions made by a float and includes either one or more
descending profiles and/or one or more ascending profiles. Typically it also includes phases of
subsurface and surface drift.
TC: OK, that definition is better; there may be more than one ascending or descending profile.
=============================================================================
§2.2.4 §2.3.4 §2.4.4 :FLOAT_SERIAL_NO dimension extended to 32 instead of 16
TC: corrected
=============================================================================
§2.2.4, §2.3.4, §2.4.4 : add a “conventions” attribute to PLATFORM_TYPE
TC: OK, done
§2.4.4 : add a “conventions” attribute to PLATFORM_FAMILY and PLATFORM_MAKER
TC: OK, done
§2.4.7.1 : add a “conventions” attribute to SENSOR_MAKER and SENSOR_MODEL
TC: OK, done
http://tinyurl.com/nwpqvp2
While realizing that this document needs regular updating I'm somewhat concerned about relying on
links to point users to crucial information. Not sure what the best way to do this might be. Maybe
make it available via the ADMT web site (instead of google docs or link from there to google docs - I
looked at the ADMT documentation site and did not see a link to the document)?
TC: we do not have a better solution now
Another problem is: the document loads awfully slow (e.g. PLATFORM_TYPE/KEY)
Also, a subset of the information seems to be in Table 23 of the user manual.
Maybe we can have pdf's instead of "goolge docs" (one for each sheet in the doc)
and link to them from the ADMT documentation page.
(I'm still waiting for PLATFORM_TYPE/KEY after typing all this.)
=============================================================================
§2.3.5 : add a “resolution attribute” to JULD and JULD_ADJUSTED
TC: done
§2.3.6: add a “conventions” attribute to each cycle timing
TC: done
in general:
1) what should a DAC do if a resolution for a variable is not available?
TC: in chapter 2.2.5.1, a sentence says “don’t put a resolution attribute”. For me, it implies that
“resolution” attribute should not exist if the resolution is not known. I added the sentence on
§2.2.5.1 : “The resolution of a parameter is reported in “resolution” attribute. If the resolution is
unknown, the attribute “resolution” is not reported.”
2) for derived quantities (like DOXY or PSAL when derived from CNDC) the
resoultion may depend on who one asks what it is.
3) 1 second resolution in terms of JULD is 1.157407407407407e-05 days if
one uses double precision. What should the resolution be set to?
1e-05 days?
4) what if the resolution depends on the profile number in a given cycle?
TC: see §2.2.5.1
=============================================================================
§2.3.5.1 How to report unusual Pressure resolutions in the
N_MEASUREMENT variable group of the TRAJ file
=============================================================================
§2.4 : general revision of metadata format; §2.4.6 adding launch
configuration parameters; §2.4.7 discriminate float sensor and
float parameter information
user_manual_version Eponymous
---> not sure Eponymous is the right word here (people I asked were baffled,
and the definitions I found online did not seem to fit)
TC: a proper definition replaces “eponymous”: “The version number of the user manual”
N_SENSOR N_SENSOR = <int value>; Number of sensors mounted on the float and used
to measure the parameters.
TC: OK
N_LAUNCH_CONFIG_PARAM N_LAUNCH_CONFIG_PARAM=<int value>; Number of pre-deployment
or launch configuration parameters.
TC: OK
LAUNCH_CONFIG_PARAMETER_NAME(N_LAUNCH_CONFIG_PARAM, STRING128)
LAUNCH_CONFIG_PARAMETER_VALUE(N_LAUNCH_CONFIG_PARAM)
----> I see a problem with this, because some floats of the park and profile type
have a dual launch configuration (e.g., profile depth alternating in some way between
2000 and 1000 dbar). So this is by definition a launch configuration, but it can not
be stored in LAUNCH_CONFIG_PARAMETER_....
TC: these are 2 conffigurations (see Esmee’s message below).
----> Unless we find a reasonable solution for this, we need to rethink the approach
of having LAUNCH_CONFIG_PARAMETER_.... as well as the CONFIG_PARAMETER_....
that come with CONFIG_MISSION_NUMBER and CONFIG_MISSION_COMMENT.
TC: I am not sure to understand the issue
=============================================================================
§2.4.5 : add STARTUP_DATE and STARTUP_DATE_QC
=============================================================================
§2.6 : B-Argo profile additional format features
§2.6.1 : manage pressure axis between core-argo and b-argo files
DOWNWELLING_IRRADIANCE_RAW:wave_length_nanometer = "115 132 149 166 183 200 217 234
251 268 285 302
319 336 353 370 387 404 421 438 455 472 489 506 523 540 557 574 591 608 625 642 659 676 693
710 727 744 761 778 795" ;
----> could there be a way to store this as a vector rather than a string in
the nc file header part? That would make it more accesible to users.
TC: maybe, I think that this exists for QC flags, should be tested.
N_VALUES41
----> N_VALUES (throughout)
=============================================================================
§3.22, §3.23, §3.24, §3.25, §3.26, §3.27 : new reference tables
§ : separation of core-Argo and B-Argo parameters in reference table 3
Is most of this content from the docs under http://tinyurl.com/nwpqvp2?
Can't check because still waiting for it to load. See comment on this above.
TC: no, the parameters list is in Argo data-management web site :
http://www.argodatamgt.org/Documentation/Draft-documents
=============================================================================
§4.1.2 : separation of core-Argo data files and B-Argo data files
"4.1.2.2 B-Argo individual merged profile file
To facilitate the use of B-Argo data, the GDAC merges each B-Argo file with its corresponding
core-Argo data file.
The merged file contains the core-Argo and B-Argo parameters listed on reference table 3. The
intermediate parameters are ignored by the merged files."
----> will this allow DACs to only submit merged files? If so: How would that impact
users?
TC: no, DACs will provide core-Argo and B-Argo files, merged files will be provided by GDAC only.
regarding the B-Argo traj file: Is this intended to only contain bio-PARAM
with CYCLE_NUMBER, JULD and MEASUREMENT_CODE, so that it can be linked to
the core traj file?
TC: yes
=============================================================================
§7 : add a glossary
TC: done
=============================================================================
§3.1 : add the B-Argo profile and B-Argo trajectory data types in reference table 1.
TC: done
=============================================================================
§3.3 : revision of valid_min and valid max on TEMP, PSAL, DOXY to be compliant
with Argo quality control manual.
Do we really want to force PSAL 2 to 41 instead of the old 0 to 42?
TC: yes
=============================================================================
§2.2.3, §2.3.3: add a FILE_TYPE variable to discriminate C, B or M files.
Do we really need this one? The file name already gives it away.
TC: no, we do not need it. This information can come from the variable “data_type”. The reference
table 1 contains 4 new data types : B-Argo profile, B-Argo trajectory B-Argo profile
B-Argo trajectory, Argo profile merged, Argo trajectory merged
=============================================================================
§2.2.6: list parameter names in the PARAMETER variables even if
no calibration is performed yet.
=============================================================================
Page 1 : add a DOI to the manual
TC: done
=============================================================================
Page 2 : add a “How to cite this document” chapter
=============================================================================
§6.5: add a chapter on Argo DOIs
TC: done
Claudia, 30/04/2014 20 :23
in addition to what I sent before I thought we may be able to consolidate the two tables in sections
2.4.7.1 and 2.4.7.2 (see attached word document).
TC: probably not necessary (see Esmee’s message below)
To make this work we would need to introduce N_PARAM2, which would be the maximum number
of parameters an individual sensor on a float can measure.
For example, if a float has an oxygen sensor that reports a phase and a temperature then
N_PARAM2=2 (or larger depending on other sensors the float may have).
Name
SENSOR
Definition
char SENSOR(N_SENSOR, STRING32);
SENSOR:long_name = "Name of the sensor
mounted on the float";
SENSOR:conventions = "Argo reference table
25";
SENSOR:_FillValue = " ";
Comment
Names of the sensors mounted on the float
Example: CTD_PRES, CTD_TEMP, CTD_CNDC,
OXYGEN_OPTODE.
See Argo reference table 25.
Regular updates are made to an online version of
this table available at:
http://tinyurl.com/nwpqvp2
SENSOR_MAKER
SENSOR_MODEL
SENSOR_SERIAL_NO
SENSOR_PARAMETER
SENSOR_PARAMETER_
UNITS
SENSOR_PARAMETER_A
CCURACY
SENSOR_PARAMETER_R
ESOLUTION
char SENSOR_MAKER(N_SENSOR,
STRING256);
SENSOR_MAKER:long_name = "Name of the
sensor manufacturer";
SENSOR_MAKER:conventions = "Argo
reference table 26";
SENSOR_MAKER:_FillValue = " ";
char SENSOR_MODEL(N_SENSOR,
STRING256);
SENSOR_MODEL:long_name = "Type of
sensor";
SENSOR_MODEL:conventions = "Argo
reference table 27";
SENSOR_MODEL:_FillValue = " ";
Name of the manufacturer of the sensor.
Example : DRUCK, SBE, AANDERAA.
See Argo reference table 26.
Regular updates are made to an online version of
this table available at:
char SENSOR_SERIAL_NO(N_SENSOR,
STRING16);
SENSOR_SERIAL_NO:long_name = "Serial
number of the sensor";
SENSOR_SERIAL_NO:_FillValue = " ";
char
SENSOR_PARAMETER(N_SENSOR,N_PARAM2,
STRING64);
:long_name = "Name of parameter computed
from float measurements”;
:conventions = "Argo reference table 3";
:_FillValue = " ";
Serial number of the sensor.
Example : 2646 036 073
char
SENSOR_PARAMETER_UNITS(N_SENSOR,N_P
ARAM, STRING16);
:long_name = "Units of accuracy and
resolution of the parameter";
:_FillValue = " ";
char
SENSOR_PARAMETERACCURACY(N_SENSOR,N
_PARAM2,STRING32);
:long_name = "Accuracy of the parameter";
PARAMETER_ACCURACY:_FillValue = " ";
char
SENSOR_PARAMETER_RESOLUTION(N_SENSO
R,N_PARAM2,STRING32);
:long_name = "Resolution of the parameter";
:_FillValue =" ";
http://tinyurl.com/nwpqvp2
Model of sensor.
Example: DRUCK, SBE41CP,
AANDERAA_OPTODE_3930.
This field is now standardised.
See Argo reference table 27.
Regular updates are made to an online version of
this table available at:
http://tinyurl.com/nwpqvp2
Names of the parameters measured by float
sensors or derived from float measurements.
The parameter names are listed in reference table
3.
Examples :
TEMP, PSAL, CNDC
TEMP : temperature in Celsius
PSAL : practical salinity in psu
CNDC : conductvity in mhos/m
Units of accuracy and resolution of the parameter.
Example : psu
Accuracy of the parameter.
Example: "8 micromole/l or 5%"
Resolution of the parameter returned by the
sensor (note that this is not necessarily equivalent
to the resolution of the parameter returned by the
float through telemetry).
Example : 0.001 micromole/l
Esmee, 01/05/2014 05:03
Some feedback on Claudia's suggested changes to the metadata manual..
I agree that "Eponymous" as a definition for the user_manual_version could be simplified. I suggest
instead: "The version number of the user manual".
TC: OK, done
The conventions attributes for PLATFORM_TYPE, PLATFORM_FAMILY, PLATFORM_MAKER,
SENSOR_MAKER and SENSOR_MODEL have already been added by Thierry in this latest version 3.1
(30/11/2013).
The reference tables for the PLATFORM and SENSOR variables will need to be constantly updated as
new types are added.
We are still working on the SENSOR and SENSOR_MODEL tables to include all of the new Bio sensor
types and making sure these are named in a consistent and sensible way.
Mathieu is maintaining these tables using google docs - I agree that it would increase accessibility to
also link to these via the ADMT website. We don't want to maintain two duplicate sets though, so
suggest adding the link to the google docs for now.
TC: ok, no change in the manual
A hard copy of the most up-to-date version of these tables is also present in the manual.
Thierry we also need to add some text under the headers of each of the reference tables (22 to 27)
to let people know that these are frequently updated and they should check the URL for the latest
table. I suggest something like "Please note that these reference tables are frequently updated to
include new sensor and float models. You can find the latest version at: http://tinyurl.com/nwpqvp2"
TC: ok, done
There is no conflict with the park and profile floats that alternate between 2000 and 1000 dbar.
These are simply different missions!
In this case, you would report the initial profile setting in the launch configuration settings and in
mission 1 and then the changed value becomes a new mission (# 2). The float would then alternate
between missions 1 and 2 on subsequent profiles.
TC: ok, no change needed
I would prefer to keep the information in tables 2.4.7.1 and 2.4.7.2 (SENSOR and PARAMETER)
separate as I think merging the two may be more confusing to users.
TC: ok
Ann, 01/05/2014 07:15
A few comments from me:
We discussed the definition of a cycle at some length at ADMT (sorry you were missing) and
concluded that it is defined as the time period from one surfacing of the float to the next. This was to
eliminate the questions of 'sub cycles' being raised for some bio floats that surface but don¹t transmit
every time they hit the surface and Jean-Philippe wanted to include all of these in one profile file.
This was considered unnecessarily complicated so we restricted the definition of a cycle to be
effectively one up and down, whether a profile was sampled in both directions or not (and each
ascending or descending profile is already in its own file now, even if they are the same cycle).
Bounce profiles, however, where the float doesn't surface, are all carried in one profile file with multi
dimensions, one for each bounce. So that eliminates the words 'one or more' from the definition. I
think Thierry's words are just fine.
Thierry, I note that the definition of FLOAT_SERIAL_NO (page 51) still has
STRING16 instead of STRING32 - which is correct?
TC: the correct definition is STRING32, corrected.
Good catch, Claudia, on eponymous! Esmee's suggestion works for me.
The tables are a real issue - and with the bio floats coming on line at an ever increasing rate, it's likely
to get worse before it gets better. For now, though the current location of the docs isn't ideal (we
should be able to find a faster place to put them), I think we're stuck with them.
Even the tech tables are still changing. However! Down the track if and when things stabilise a bit,
the tables physically in the manual might actually be current.
N_VALUES41 is necessary, I'm afraid. The bio floats can report different numbers of wavelengths, for
instance, for different variables so, for instance, they could report the full set (256 values for EACH
PRESSURE LEVEL!) if they wanted to, or they might report 41 for nitrate calculations (41 for each
pressure level) and then they might report a different set of 32 wavelengths for bisulphate
calculations (you would then declare N_LEVELS31=31). This avoids declaring N_LEVELS for the
biggest dimension (256 or 41) and then have lots of (more ) wasted space in the files. I know it's not
ideal but it's up to the bio people to manage this, not us so it will only be an issue when you get bio
funding and have a dedicated person working with this data.
In answer to your question: ----> will this allow DACs to only submit merged files? If so: How would
that impact users?
NO - DACS will only be able to submit the separate files in my opinion, since that is the ONLY way to
get the raw parameters into the system and available globally. If DACs submit only merged files,
then the data used to calculate DOXY of CHLA would be 'lost'. All three files, C, B and M, will need to
be available at the GDACs.
TC: yes I agree, no change needed in the manual
And your answer to the next question is yes, B traj files will contain the B variables and raw pressure
only - in other regards, they will be identical with the C traj files (and depending on how/when the
drift bio data is measured, they might have some duplication). I also think if no bio data is measured
during drift or during a part of the cycle reported in the taj files, then it isn't necessary to create a B
traj file - John?
Can you confirm this?
Thierry, Esmee and I both agree with Claudia that FILE_TYPE isn't necessary. Where did this come
from and is there a valid reason to include this?
TC: some applications such as the file format checker needs to identify the type of file: C: core
parameters data file; B: non core parameters data file; M: core and non-core parameters merged
data file. Depending on the file type, the controls are different (such as the allowed parameters).
However, we do not need the new variable FILE_TYPE. This information can come from the
variable “data_type”. The reference table 1 contains 4 new data types: B-Argo profile, B-Argo
trajectory B-Argo profile, B-Argo trajectory, Argo profile merged, Argo trajectory merged
That seems to be all I can comment onŠ Thanks for taking such a close look, Claudia!
Claudia, 01/05/2014 16:37
thanks for the clarifications. What still is not quite clear (to me at least) follows below.
> We discussed the definition of a cycle at some length at ADMT (sorry you > were missing) and
concluded that it is defined as the time period from one > surfacing of the float to the next. This was
to eliminate the questions of > 'sub cycles' being raised for some bio floats that surface but don¹t >
transmit every time they hit the surface and Jean-Philippe wanted to > include all of these in one
profile file. This was considered > unnecessarily complicated so we restricted the definition of a
cycle to be > effectively one up and down, whether a profile was sampled in both > directions or not
(and each ascending or descending profile is already in > its own file now, even if they are the same
cycle). Bounce profiles, > however, where the float doesn't surface, are all carried in one profile >
file with multi dimensions, one for each bounce. So that eliminates the
> words 'one or more' from the definition. I think Thierry's words are
> just fine.
I'm not sure I understand what you mean. I'm not trying to introduce sub-cylces, all I meant is:
bouncing floats have up to 7 ascending profiles in one cycle. If the text limits the number of
ascending and descending profiles to one each within a cycle, then users will wonder about 7
ascending profiles in the same cycle.
With the text I proposed this 'surprise' can (maybe) be avoided.
(A cycle is defined as a series of actions made by a float and includes either one or more descending
profiles and/or one or more ascending profiles. Typically it also includes phases of subsurface and
surface drift.)
> N_VALUES41 is necessary, I'm afraid. The bio floats can report different > numbers of
wavelengths, for instance, for different variables so, for > instance, they could report the full set
(256 values for EACH PRESSURE > LEVEL!) if they wanted to, or they might report 41 for nitrate >
calculations (41 for each pressure level) and then they might report a > different set of 32
wavelengths for bisulphate calculations (you would > then declare N_LEVELS31=31). This avoids
declaring N_LEVELS for the > biggest dimension (256 or 41) and then have lots of (more ) wasted
space > in the files. I know it's not ideal but it's up to the bio people to > manage this, not us so it
will only be an issue when you get bio funding > and have a dedicated person working with this data.
I know it is needed, all I wondered about is that it is called
N_VALUES41 and not N_VALUES. Who is to say it will always be 41?
TC: see John’s answer below
>
> In answer to your question: ----> will this allow DACs to only submit > merged files? If so: How
would that impact users?
>
> NO - DACS will only be able to submit the separate files in my opinion, > since that is the ONLY
way to get the raw parameters into the system and > available globally. If DACs submit only merged
files, then the data used > to calculate DOXY of CHLA would be 'lost'. All three files, C, B and M, >
will need to be available at the GDACs.
This does not solve our problem (no funding for bio-Argo). Sylvie or Thierry: maybe you can call me
so we can discuss how to proceed.
> And your answer to the next question is yes, B traj files will contain the > B variables and raw
pressure only - in other regards, they will be > identical with the C traj files (and depending on
how/when the drift bio > data is measured, they might have some duplication). I also think if no >
bio data is measured during drift or during a part of the cycle reported > in the taj files, then it isn't
necessary to create a B traj file - John?
> Can you confirm this?
If the intent is that B-traj file is the C-traj file plus the bio-Argo fields minus the core Argo PARAM
fields, then DM operators for traj files have to clean up all the position and cycle information in two
files. That seems like an unnecessary complication of their job to me.
TC: for the moment, that is the situation of traj-file that will be managed like profile files : core, bio
and merged files.
John, 01/05/2014 19:38
Argo float cycle definition:
What the manual currently reads...
"It ends with the next ascent to shallow water and data transmission (in some situations or for some
floats, data transmission may not always occur)"
I recall a slightly different discussion at the ADMT. I thought it was agreed that a cycle was defined
via a "planned" ascent to the surface.
Thus if a float is programmed to attempt to go to the surface, it is a cycle. Thus a float that goes to
the surface but does not transmit is still a cycle. A float that attempts to go to the surface but can't
due to ice, is still a cycle. A float that does not attempt to go to the surface but does an
intermediate, mid-depth profile (is this the case with bounce floats?), would not have to declare a
new cycle number. I think the word "planned" or "programmed" is crucial to this definition.
Also If we are defining the start of a cycle when it "descents towards deep water" (usually MC100 for
all of you trajectory measurement code people), then we must be more clear with the 'end' of a
cycle. If we define the 'start' as above, then a cycle ends after the FULL surface interval is complete,
which would of course include the transmission phase. But there can be significant time on the
surface after transmission.
I suggest, replacing the above, "It ends after the next programmed ascent to the surface, and if
begun, after the full surface interval has been completed. During the surface interval, data
transmission typically occurs but it is not a requirement for a cycle to have occurred."
Hopefully this clarifies the Bounce Floats and other types, as well as clarifies the use of cycle number
in the trajectory file.
TC: ok, the cycle definition is updated in the manual.
N_VALUES41:
I agree Claudia that it is ugly...but all choices are ugly. It won't always be N_VALUES41. Only when
there are 41 values. There, for example, may be a N_VALUES41 and an N_VALUES36, etc, in the
same file. Thus the user must parse the dimension name. Everyone is very worried about data file
size, and just making N_VALUE be the maximum necessary for a given file would make the data file
sizes larger.
TC : OK for me.
B-traj and C-traj:
Ann, I concur that there is no reason to create a B-traj file if there is no BIO data collected in float
phases not reported in the profile file.
That being said, what we haven't really defined strongly is what data goes in which file. For example
we now have unpumped NST that can extend onto the surface (P not monotonic) for a 'short' while.
This data goes into the profile file currently, and we change the Real-Time tests to allow that.
But what is 'short'? We can't determine every possibility now. I bring the topic up simply to say that
there is grey area. And if the overhead to creating a B-traj (and for that matter the C-traj) file is high
then there will be movement to place as much as possible in the profile file.
Claudia, I concur that splitting the B-traj and the C-traj has transferred lots of potential work from the
DAC to the DMQC operator. Let me just say that I believe that as the decision has been made to split
BIO and Core data, it should be done in all files. Going forward we will need feedback from the B-traj
DMQC community, on how best to handle the B-traj files. The issue you raise is a great one, if there
is a change in position/timing or other info in the C-traj file during DMQC, how does the B-traj file
digest any modifications and propagate it into their file. Time and position are the fundamental
information of the traj file, so it would be difficult to leave out of the B-traj. I don't have any great
suggestions, but if the B-traj community develops a system I don't see why the B-traj has to mirror
the C-traj.
TC: ok, the manual remains unchanged
Claudia, 01/05/2014 19:47
Just to clarify what the bounce floats do:
They transmit their cycle number.
Odd numbered cycles are normal cycles.
Even numbered cycles are "bounce cycles".
A normal cycle takes several days. A bounce cycle takes less than a day.
TC: no impact in the manual
Claudia, 01/05/2014 20:39
> Thanks Claudia, although you don't specify whether a bounce takes it to > the surface. What I
state below assumes no, but I've been mis-informed > about bounce cycles before. I believe that for
bounce floats....trying > to tie everything together....
sorry about that. You are right, they do not reach the surface.
> Upon deployment, LAUNCH_CONFIG_PARAMETER_<> would be filled with 'normal > cycle'
parameters (cycle 1) and CONFIG_PARAMETER_NUMBER=1 would contain > similar to the
LAUNCH_CONFIG_PARAMETER. CONFIG_PARAMETER_NUMBER=2 would > be the bounce
configuration. This probably should include the number of > bounces, since all repeated bounces
are considered a single 'mission'.
> If the number of bounces changes in a cycle then you must introduce a > 3rd (or more)
CONFIG_PARAMETER_NUMBER with the variation in 'number of > bounces'.
>
> Then CONFIG_MISSION_NUMBER would alternate between 1 and 2 (i.e.
> 12121212121...), and the float reported cycle number could be retained.
Fine with me. However, there is one thing we need to be aware of:
if a float somehow messes up the data transmission, it may look like it changed the configuration. We
will not treat this as a new mission number.
> All ascending profiles (if they are taken) will be in a single cycle > file in different N_PROF (and that
cycle number will be as reported by > the float). Odd cycle numbers would have N_PROF=1, and
even cycle > numbers would have N_PROF=7.
Yes.
> All descending profiles (if they are taken) will be in a single cycle > file (labeled with a 'D' at the
end of the filename) in different N_PROF.
So far, I'm not aware of bounce floats with descending profile cycles.
> NOTE: I still really dislike the ascending and descending file split, > but C'est la vie.
>
>
> But let's look at it in the case if the float does purposefully 'bounce'
> to the surface, then things change. Whatever the float reports for > cycle number is immaterial
and the decoder must apply a calculated cycle > number.
For now, this is a hypothetical float (or???, we do not have such floats). Some thoughts:
1) If it purposefully bounces to the surface, then it is likely to transmit data.
2) At what pressure would you consider that the float is at the surface?
3) Why would the cycle number transmitted by a float be changed to something else, just because it
bounces to a pressure shallower than X dbar? If a bounce floats we have went above X dbar it might
end up having a normal cycle 1, 2-8 might be bounce cycles and then the float surfaces for the next
normal cycle (which it numbers as 3) and we would have to change that to 9. Going to the extreme
with this: if a couple of bad pressures are received from a bounce float, it may look as if it is at the
surface, but in reality it did its routine bouncing below the surface.
Because of the difficulty in deciding if the float surfaces while bouncing, and because of the can of
worms the manipulation of the cycle numbers opens, I'm not a fan of the method suggested below.
Claudia
> Upon deployment, LAUNCH_CONFIG_PARAMETER_<> would be filled with 'normal > cycle'
parameters and CONFIG_PARAMETER_NUMBER=1 would contain similar to > the
LAUNCH_CONFIG_PARAMETER. CONFIG_PARAMETER_NUMBER=2 would be a > bounce
configuration. Since each single bounce is a cycle (because it > goes to the surface) there would be
no 'number of bounces' CONFIG.
>
> Then CONFIG_MISSION_NUMBER would alternate between 1 and 2 in a > different pattern (i.e.
12222222122222221...), assuming 7 bounce cycles > in between each normal cycle.
>
> In each profile file there would be only 1 N_PROF (although possibly two > files, descending and
ascending), and its cycle number would be > calculated during parsing.
>
> Hopefully my understanding of the META file is correct, > John
John, 01/05/2014 21:04
It's always good to come up with future possibilities and 'what ifs'.
First, if I recall the discussion at the ADMT, it was decided that Argo was not locked into a
manufacturer's defined cycle number, if it differed from ours. Thus we by default are allowing for
the possibility of not reporting the cycle number transmitted by the float through the Argo system.
Hopefully this won't happen often.
> Fine with me. However, there is one thing we need to be aware of:
> if a float somehow messes up the data transmission, it may look > like it changed the
configuration. We will not treat this as a > new mission number.
Correct you will not treat it as a new mission number. Meta data is what you expect, it has nothing
to do with what you get. If the float behaves differently than expected, it's mission is still unchanged.
> For now, this is a hypothetical float (or???, we do not have such > floats). Some thoughts:
> 1) If it purposefully bounces to the surface, then it is likely to > transmit data.
But it might not. If I recall the floats Thierry discussed, they went to the surface and did not report.
> 2) At what pressure would you consider that the float is at the > surface?
Very good question. I would suggest 5 dbar or shallower. I would think the number could be slightly
deeper, due to the difficulty in trying to control buoyancy very close to the surface. Our ice detection
software looks at T between 40 dbar and 20 dbar to decide whether to abort surfacing.
So it is possible to 'bounce' from as shallow as 20 dbar (although you'll have to be controlling rise
speed to do it).
> 3) Why would the cycle number transmitted by a float be changed
> to something else, just because it bounces to a pressure shallower
> than X dbar? If a bounce floats we have went above X dbar it
> might end up having a normal cycle 1, 2-8 might be bounce cycles
> and then the float surfaces for the next normal cycle (which it
> numbers as 3) and we would have to change that to 9. Going to the
> extreme with this: if a couple of bad pressures are received from
> a bounce float, it may look as if it is at the surface, but in
> reality it did its routine bouncing below the surface.
As I mentioned earlier, the data received by the float has no determination in the metafile or
determined cycle number. As an example, every once in a
while a SOLOII pump hiccups and after diving to 100dbar goes straight back to the surface. If it
managed to get off the surface, I would not declare
that 'odd' surfacing a new cycle. Yes it did it, but it wasn't intended. Its CONFIG did not change. If a
bounce float just happens to get above X
decibars but it was targeting deeper then nothing changes with the metafile.
I would turn around your question, why wouldn't we change the cycle number to something else, if
the float reported a cycle number very different then
what Argo wants? This has already occurred. To try to get manufacturers to follow the definition of
cycle number that we prefer, Argo needed to
define what it means to be a cycle. That is what we've tried to do. Hopefully manufacturers will fall
follow what is recommended.
Ann, 02/05/2014 00:54
Absolutely correct, John - glad your recollection is more precise than mine. I forgot the 'ice' nuances.
And also glad you responded on the traj question - I am out of my depth generally on those. A lot of
these questions can only be answered when the bio people begin to actually code and use these files
and we see how hairy they are. I expect lots of discussion into the next 5 years at least ;-)
Ann, 02/05/2014 00:58
I thought bounce profiles were a series of profiles performed between full profiles - i.e., they did a
full depth profile and to the surface and then did, for instance, 3 profiles fast and between 1000 and
200m - but I've never used one of these, nor seen their data. That's just how I remember their
behaviour being described originally and why we needed to consider putting them into one cycle,
different profile dimension, in the same file. If what you say is how the float behaves, then they're
irrelevant for this discussion because they would each be reported with a different cycle number, in a
different file, and with a different mission number.
Thanks for clarifying this!
Claudia, 02/05/2014 17:17
I'm sorry, but I'm not sure I understand "they would each be reported with a different cycle number,
in a different file, and with a different mission number"
because
I'm not sure what "they" refers to.
Anyway, just to avoid any potential for confusion:
cycle 1: 1 deep profile
cycle 2: 7 short bounce profiles
cycle 3: 1 deep profile
and so on.
Ann, 05/05/2014 03:11
Given the discussion that's surrounded this, and assuming John is correct that the float doesn't aim
for the surface when doing the short bounce profile, then you're correct - and true, my use of 'they'
is confusing - in the start, I meant the float and at the end, the profiles! Sorry - blame haste.
To summarise, you would have 3 'core' profile files for a float that performs this behaviour, with two
missions. The second profile file would contain 7 profiles and all of the bounce data. Does that
match your understanding?
Ann, 06/05/2014 03:34
Sorry this is a bit out of sequence but I wanted to check that we had these things sorted out.
Are you clear on cycle/bounce profiles now? We had to be very careful because some bio floats
come to the surface but transmit only every 7th or so cycle which the called 'sub cycles' and it got
really messy. This required a more restrictive definition of cycle and the bio people will need to
'construct' cycle numbers of these, putting each in a separate file. The bounce profiles, however, are
true subsidiary cycles and all go in one file.
I assume John's answer to the question of N_LEVELS## works? Again, without bio floats, we'd all be
laughing! ;-)
If I were you, and didn't have any funding for the bio data, (ignoring oxygen - do you have funding for
that?), I'd just not report that data and if your PIs want it, they would have to provide the resources
to get it into the system. To some extent, that's the approach we've taken - I don’t have the
conversion routines for flbb data, for instance and though it's simple, don't have time to code it. So
I'm not putting the derived variables in the files until my PIs get to it, though I am adding the raw
parameters. That is one way to put pressure on the bio people to actually contribute to the data
management.
I think the conclusion of ADMT14 was that the bio people are responsible for the complexity of the
system we are forced to deal with and so they need to bear the brunt of whatever duplication or
extra work is created.
So the fact they will need to clean up positions or timing information in a bio-traj file is not
something that the core people need to worry about.
In many regards, simply expanding the core files to include bio data would have been the easy
solution - but would lead to severe resource issues with people who are funded only to handle core
data. By splitting the files and so splitting the responsibility, I think we've come to the only sensible
solution and the complexity then needs to be dealt with by those who are best funded for it. There is
no question that once these files are separated, the job of the core people will be much easier.
Ann, 06/05/201403:50
Another 2 cents worth. I disagree with John - I am not sure we can actually define a depth where a
float 'surfaces'. I think two things are at work here - intent and execution. So under ice profiles, for
example don¹t hit the surface and may not get even close to 5 db but are still independent cycles
because they were aiming for the surface. Similarly with some of our older floats who can't surface
when the surface waters are too light - they are still cycles. But something that intentionally stops
before it surfaces (10db? Maybe even 5?) even if an error occurs and it accidentally hits the surface,
on a float performing a 'bounce'
mission for example, would be treated very differently and all of the bounce profiles would occur in
one cycle file.
The complexity arose because of some bio floats that perform what they call 'sub cycles' - they
surface but don't transmit and count all of those as one cycle. We would then end up with one file
with 7 profiles that are indistinguishable from the primary profiles of other files and it was decided
hiding these in a secondary dimension was overly complex and might confuse users. Basically, if it
acts like a primary profile, looks like a primary profile and quacks like a primary profile then it should
be treated as primary profile, whether or not it transmits :-). This has been accepted so we move on.
And I agree with John - I really hope manufacturers take the hint and start to realise that simplicity is
what we need (not to mention consistency! And un-changing formats without a good reason! And
not adding layers and layers to float behaviour without a good reason! AndŠ..!)
Claudia, 06/05/2014 15:04
bounce floats: OK
N_LEVELSnnn - OK, the choice of name is a bit odd, but I guess it mirrors the STRINGnnn pattern.
cycle numbering topic: I still think it is beneficial to stick with the cycle numbers assigned by the float
as it operates. If a float surfaces several times within a cycle it could go into increasing NPROFs within
a single cycle file, because each NPROF comes with its own slot for times and location (allowing for
providing GPS fixes if these are done in those bio-floats). For such floats
NPROF=7 would mean the float did 7 subccyles. If such a float reached the surface can be
determined by looking at the pressure in the profile files.
In the trajectory files it can be determined by looking at the measurement code. How a float like that
behaves would also need to be recorded in the mission configuration in the meta file.
Mathieu, 06/05/2014 15:20
Hi all,
You can add the links on ADMT website as follow:
(Dynamic) Reference Tables for PLATFORM, SENSOR, AND DATA_FORMATS
https://docs.google.com/spreadsheet/ccc?key=0AitL8e3zpeffdEtyVmN3a0hvUC1NMDJMcHlLN2FMSl
E&usp=drive_web#gid=1
or
http://tinyurl.com/nwpqvp2
and
https://docs.google.com/spreadsheet/ccc?key=0AitL8e3zpeffdENUQmszRlY3djYweGZhbnBZSU1fTFE
&usp=drive_web#gid=9
or
http://tinyurl.com/mf5wcvd
Claudia, 15/05/2014 09:25
I thought I had asked it before, but I looked for an email about it without success, so here it comes...
Under the global attribute "featureType" for the profile file it says it should be filled with
"trajectoryProfile". Is this a typo, or is this intentional?
TC : The feature type trajectoryProfile is a series of profile features located at points ordered along
a trajectory. I think it is relevant for Argo NetCDF profile files.
Source : http://cfconventions.org/1.6.html#idp6280704
Comments on version 3.0 toward version 3.01, December 2013
Ann, 20/02/2014 23:03
oxygen discussion
Remember that people (we) are producing version 3.0 files right now that contain both oxygen and
the raw variables. These are perfectly acceptable and will have to be until DACs get around to
separating them. And though CSIRO will be transitioning to the B/C files in the next few months or
less if I can sort out the formats completely, I don't see an issue with Claudia doing that for a little bit
of time (but not forever).
And as I started to say in another reply (not sent), DACs where oxygen undergoes DMQC separate
from the core DMQC people will have a real issue with file divergence, particularly since the core
profiles might be revisited many times. So we need to have all bio variables (including
oxygen) in the B files and as soon as possible. It really is a pretty simple coding task to produce
separate files (I'm duplicating my 3.0 code in two scripts that generate the netcdf and then just
commenting out the bits that no longer apply - not quite that easy but you get the drift).
So it's possible that coding the separate 3.0 files for Claudia will turn out to be as simple as coding a
combined 3.0 file. Just a thoughtŠ
Claudia, 20/02/2014 15:09
oxygen discussion
Summary of the messages below regarding contents of core files
(R-files) of floats that measure oxygen in one or more profiles during a cycle:
- the primary profile is the core Argo profile (which may or
may not have oxygen)
---- I think everybody agrees on this one. ---- the secondary profile of a multi-profile oxygen float may
have interpolated or measured T and S, or a combination of
these. This profile will go in NPROF=2.
---- I think everybody agrees on this one. ---- allow oxygen measurements in the 3.0 R-file for a transition
period, without all the raw data, for DACs that do not have
the resources to create B-files in for an unknown period of
time.
---- Is this how we want to proceed? ----
- once a DAC can create the B-files, the oxygen could be pulled
out of the R (or D) file and included in the B-file.
---- Is this how we want to proceed? ----
Thierry, 21/02/2014 12:01
The B-Argo files are described on page 63, §2.6 “B-Argo profile and trajectory format additional
features”.
I added a chapter §2.6.1 “Pressure axis management in profile files”. Can you correct/enhance it:
The same pressure vertical axes PRES are shared between core-Argo and b-Argo files.
A core-Argo file and a b-Argo file have the same N_PROF dimension and the same PRES vertical axis.
PRES is the simple, strong and unambiguous link between core and b-profiles.
Some pressure profiles of the core and b files may be alone:
•
case 1: one b-profile without core parameter will have a pressure only profile in the core-file
•
case 2 : one core-profile without b-parameter will have a pressure only profile in the b-file
PRES is the only variable duplicated in core-Argo and b-Argo files.
A core-Argo file contains adjusted pressures (PRES_ADJUSTED). The adjusted pressure are not
duplicated in the b-Argo file.
On page 71, §3.1 “Reference table 1: data type” , there are already 2 additional items:
•
B-Argo profile
•
B-Argo trajectory
The GDAC already receives the 3.0 profile files.
We need to add a new 3.1 profile format, identical to 3.0 but with only PRES, TEMP, PSAL, CNDC
(,DOXY?) allowed parameters.
This is a trivial change on GDAC.
The file checker has be enhanced to accept “B-Argo profile” and “B-Argo trajectory”. The merger
have to be developed. It may take one month to implement once we decide to do it.
Even if b and m files management is not available, the GDAC will manage the 3.1 core-files.
Thierry, 20/02/2014 12:03
Option 1 would be:
A core-Argo file contains core-Argo parameters: temperature, salinity, conductivity, oxygen (TEMP,
PSAL, CNDC, DOXY).
The intermediate parameters (temp_doxy, bphase,...) are stored in the b-file.
In your case Claudia, these intermediate parameters will not be available until you manage b-files.
I am fine with that. Is it correct for you?
Two remarks:
1. The delayed-mode on this file may be performed by different groups : the salinity d-moders and
the oxygen d-moders.
They will have to coordinate their operations, first salinity d-moding, then oxygen d-moding. This is
manageable for Coriolis DAC.
2. I think that I agree with your last question.
If N_PROF is the same in the core-file and the b-file, there is a simple, strong and unambiguous link
between b and c profiles.
Some pressure profiles of the b or c files may be alone:
. case 1: one b-profile without core parameter will have a pressure only profile in the core-file
. case 2 : one core-profile without b-parameter will have a pressure only profile in the b-file
As Justin mentioned, the first profile of the core-file will always be a regular CTD profile, with at least
PRES and TEMP.
A user may be disturbed by the PRES only profile in the second or next profiles of the core-file, or in
the b-file.
A software working with the existing 2.0 files would not crash on the 3.0 files.
Thierry, 20/02/2014 08:02
Many thanks Ann for your clear message about PRES_ADJUSTED parameter that should not be
present in the b-file.
As Justin said, there are profiles of the b-file that do not have temperature and salinity, for good or
bad reasons.
But anyway, they do have a pressure profile which is a core data.
So I like Justin’s option 1: c-file and b-file have the same N_PROF dimension.
The link between the c-profiles and b-profiles is set by N_PROF (no need for an additional variable
PRES_CoreProfileNumber).
But, data users may not appreciate a PRES only profile in the c-file…
Justin, 20/02/2014 00:00
The matching of raw pressure is not an issue. The complication in this equation is the NPROF
dimension where there are profiles that have no core data e.g. user cases such as irradiance sensors
run at high resolution near the surface or backscatter sensors near the sea bed.
I see three potential options:
1 - expand the NPROF dimension in the core file to match that of the B file even if there are no PSAL,
CNDC, or TEMP data for some of the NPROF levels; this means the pressure correction is handled
only in the core file but data users may not appreciate the apparently null data.
2 - allow PRES_ADJUSTED to be stored in the b file for the NPROF levels that do not contain core
variables; this may make the merge at GDAC level more straight forward and would not confuse
people that are only interested in the core file.
3 - potentially duplicate PRES_ADJUSTED for some NPROF levels in both b and core files.
Pressure is a special channel because it is a coordinate so an exception may be justifiable. The
delayed mode pressure correction is a relatively automated process once the sensor behaviour has
been diagnosed so should not be hard for bio operators to do.
Ann, 19/02/2014 23:05
Whoa – Thierry, you say
The B-Argo file contains non-core profiles (any parameter except TEMP, PSAL, CNDC) with their
associated PRES and/or PRES_ADJUSTED profile.
But the B files contain only PRES, I believe – correct? Or am I completely wrong? The whole point
was to make it so the core people don't need to do anything with the B files except generate it. From
then on, it's the responsibility of the Bio people to make sure they are using the latest version of the
core data when they use the bio data. We definitely do not want to duplicate the CTD data in the B
files because we then risk divergence as the core people DM the core (PRES) data but the B people
don't update the B files.
The merged files are a different matter – these will have the best current copy of all data – DM data
if available and the best derived bio data from the B files.
To set out how I thought we decided to make it work:
Core file – contains ALL CTD profiles with adjusted variables for P T and S, including the secondary
profile data for CTD – P T and S when present (or just P and T if that's all that's been reported).
B file – contains only PRES and only for the axes where bio data exists so if you have only bio data on
the subsampled pressure axis, then the primary profile disappears from the B file (this can be argued
either way but why waste the space in what is likely to be a large file for something the bio people
can't use?). But if you have full resolution oxygen, for example, and subsampled FLBB data, then the
B file contains raw PRES for the primary profile, the full res O2 data, and the raw PRES for the
secondary profile with the FLBB data.
The M file then contains the full res CTD (P T S) and Oxygen data in the primary axis, the secondary P
T and S data along with the subsampled bio data (derived variables only) in the secondary profile, all
with adjusted fields.
People who want to work with the raw bio data (or the GDACs that need to merge the files) will need
to match the axes using the raw pressure variable and then obtain the adjusted fields from the core
file which IS THE ONLY PLACE THEY WILL OCCUR (unless you use the merged file, of course).
We discussed this at the meeting and the raw pressure is sufficient to easily match the B data to the
C data (remember that the raw pressures should never change once recorded). I don't see any need
for any further N_Prof matching information.
What is really confusing is where you would EVER use a value of 0 (both examples 2) since bio data is
useless without the core data (whether on the first or secondary axis) and CTD measurements (at
least P and T?) should always exist for the bio data. Even if the values are from a secondary CTD
(PRES2, TEMP2, PSAL2) they have to exist for the bio data to be meaningful. I think this just makes
things harder to explain and understand and further makes the B and C files diverge. And further, if
for some reason we decide to include the raw pressure from the primary profile in the B files (see
above), even if there is no corresponding bio data, then all axes line up and you again, don't need this
extra bit of information.
Sorry if I'm confused by the issue… I talked to Justin about this sort of thing a couple of weeks ago
and he set me straight but it seems to be going pear shaped again!
Thierry, 19/02/2014 22 :34
Some thoughts about pressure axis:
The Core-Argo file contains profiles (TEMP, PSAL, CNDC) with their associated PRES and/or
PRES_ADJUSTED profile.
The B-Argo file contains non-core profiles (any parameter except TEMP, PSAL, CNDC) with their
associated PRES and/or PRES_ADJUSTED profile.
The B-Argo file creator will have to manage the pressure correction.
But, can the pressure correction come from the delayed mode operator ? If so, … we should maybe
insert in the Core-Argo all the pressure profiles (with pressure only profiles if they were not sampled
along the CTD).
A tricky issue for the GDACs will be to merge the C-Argo and B-Argo file into a M-Argo file (Merged).
1. In the B-file, a profile sampled along the CTD will be merged with one of the Core-Argo profile.
2. In the B-file, a profile not sampled along the CTD will not be merged with one of the Core-Argo
profile, it will appear in the M-Argo file with its own pressure axis.
So I propose that the B-file contains an additional variable for each pressure profile.
PRES_CoreProfileNumber : an integer
1. It contains the “N_PROF” value of its associated Core-Argo profile.
2. A value of “0” means that there is no merge for this profile with a core profile. It will appear as it is
in the M-Argo file.
This is not really elegant, but it provides a strict link between pressure axes of the B-Argo files with
the pressure axes of the Core-Argo files.
Is there a better option, do I miss something ?
Claudia Schmid, 20/12/2013 16:00
Questions and suggestions follow (hopefully they'll be helpful).
Question about the bio-trajectory files:
What's the concept for these (the conclusions document only mentions that "A similar strategy will
be applied to trajectory files."? I kind of hope this file is not going to be a complete traj file where
P,T,S,C are replaced by the bio-Argo PARAM, because that just seems like duplicating lots of data in
two files (and thus will generate extra work for DM operators).
Maybe, cycle number, JULD and the PARAM (plus QC, and related fields) is sufficient for bio-trajfiles?
Aside from the sentence with the word "merged" (p.71 aka p.6) I found nothing indicating what the
M-file might be.
Are we sure 99999 is a suitable fill value for all bio-argo PARAM?
"The existing data_mode variable is not related to a specific parameter. In core-Argo data files, a
data_mode = ‘D’ describes a file with adjusted salinity."
->
"For core-Argo the data_mode variable is not related to a specific parameter, a data_mode = ‘D’
describes a file in which delayed-mode quality control has been applied to all PARAM (i.e. adjustment
of salinity in conjunction with DMQC of all other PARAM)."
"In B-Argo data files, delayed mode adjustment is performed on additional parameters such as
oxygen. When relevant add a data mode related to each parameters of the file."
->
"In B-Argo data files, delayed mode adjustment may be performed one PARAM at a time. Therefore,
data_mode related to each PARAM of the file can be set independently."
Non-bio-Argo: "Note on resolution
For each parameter, the resolution attribute is mandatory. However, the resolution value is sensor
dependant.":
- replace dependant with dependent
- the resolution can also be depth dependent
Comments on version 3.0 toward version 3.01, May-June 2013
Ann Thresher, 09/05/2013
HI, Thierry – I note in the new version of the user's manual that string10 is not in the dimension list
for the profile file but is used for the firmware version – so we use string10 or string16 and pad? I'm
finalising the multi-profile file generation now so can I send you a sample when done for approval?
Thanks!! More soon…
Thierry: 15/05/2013
I propose to replace all string 10 by string 16.
String10 was only used for firmware_version, that becomes a string16.
Ann Thresher, 09/05/2013
Quick question (and there will probably be more?): in Station parameters for an oxygen float that
subsamples oxygen, the station parameter list would have 2 profile dimensions – does it have oxygen
listed in the first profile (note that it's not reported here, only in the second)? I suspect not but my
code does put it in. I think I'll change it and can always change it back after I hear from you…
STATION_PARAMETERS : List of parameters contained in this profile.
The oxygen parameter is not present in the first profile, so it should not appear in
station_parameter for this profile. It will appear in the station_parameter of the second profile.
Is that correct?
Ann Thresher, 09/05/2013
Table 8 doesn't have either the MRV Solo2 floats, nor the Seabird Navis floats – what numbers can I
use for these in the profile file instrument type field? I believe we decided that manufacturer was
irrelevant for the float types – or not? I can use 853 for the MRV Solo2 floats if that was the decision,
otherwise, we need a new code. I think these have been applied for but it's taking far too long to get
numbers approved. How do we solve this one? I'm following up with Mathieu and Etienne but
suspect the hold-up is at WMO level… Thanks!!!
I added 10 new entries in the table, based on Mathieu’s list “New Argo reference tables”.
https://docs.google.com/spreadsheet/ccc?key=0AitL8e3zpeffdEtyVmN3a0hvUC1NMDJMcHlLN2F
MSlE#gid=9
Esmée Vanwijk, 15/05/2013
Thank you for sending through version3.0 of the Argo User’s manual and apologies for the delay in
reply.
I noticed that some of the changes I sent through last year were added but there are still a few
missing?
I have re-attached the original version of the manual with the edits in track changes (some of these
are in the new manual but not all of them).
I’ve listed a summary of the major changes that still need to be made, below (these are all in track
changes in the attached word doc):
1.
Need to add CONFIG_MISSION_NUMBER in the profile file, note the FillVal = 99999
OK, I added it to the profile
2.
Replace INST_REFERENCE with FLOAT_SERIAL_NO wherever this occurs.
OK done
3.
VERTICAL_SAMPLING_SCHEME should be dimensioned by N_PARAM;
No, it remains dimensioned N_PROF, STRING256
4.
Remove SAMPLING_MODE (after discussion at ADMT – this variable causes some difficulties
for multi-profile floats as this could be ‘continuous’ for the primary profile and ‘spot’ for the
secondary profile - as it is similar to VERTICAL_SAMPLING_SCHEME we decided not to have this in
the metafile and keep it simple).
OK, removed
5.
In metadata files, in the dimensions section, add STRING128.
OK, added
6.
IMEI needs to be deleted from metafiles
OK removed
7.
Comment for TRANS_SYSTEM_ID needs updating
OK done
8.
TRANS_FREQUENCY should be dimension by N_TRANS_SYSTEM
OK done
9.
Update comment for STANDARD_FORMAT_ID and DAC_FORMAT_ID
OK done
10.
Delete ARGO_GROUP from metafile (after discussion at AST in NZ).
OK done
11.
Update the examples in the comments to FLOAT_OWNER and CUSTOMISATION
OK done for float owner
I do not have an example in mind for CUSTOMISATION
12.
CONFIG_PARAMETER_VALUE in metafile needs to be double and the FillVal = 99999
OK done
13.
We’ve deleted BATTERY_PACKS from mandatory metadata parameters and moved it to
highly desirable.
OK done
14.
Delete reference table 17 Argo group (no longer needed).
OK done
Ann or others – have I missed anything here that you can see?
From: John Gilson [mailto:jgilson@ucsd.edu]
Sent: Thursday, 16 May 2013 3:58 AM
To: argo-dm@jcommops.org
Subject: Re: [argo-dm] new version of the users manual
Hi Esmee, I agree with Marek's suggestion of STRING16 for FIRMWARE_VERSION. That would be
consistent with the trajectory file, and I hope the metafile could also be moved to String16.
OK done
In the document you sent, it lists CONFIG_MISSION_NUMBER dimensioned as (N_MISSION). This is
a typo....isn't it? I don't see a reason for the N_MISSION dimension in the profile file.
OK, N_MISSION is replaced by N_PROF
From Annie Wong awong@ocean.washington.edu, 16/05/2013 04:15
I spent some time today going through the new Version 3 of the User's Manual.
There are two issues I'd like to raise.
First, PRES_DOXY in Reference Table 3 should now be removed, or at least noted that it is now
obsolete with the new single-cycle multiple vertical sampling scheme format. Also, there are several
new _DOXY variables that need to be added to Table 3 (TPHASE_, C1PHASE_, C2PHASE_, MLPL_, etc).
Please consult with Virginie.
OK, PRES_DOXY is removed from table 3 and I consulted Virginie for *_DOXY variables.
Second, a fairly major point regarding the variables in the single-cycle files.
As I noted in my reply to Esmee's email yesterday, I had only just noticed that the N_PROF dimension
had been added to all the variables in Section 2.2.4.
It dawned on me today that perhaps not many people are aware of this, because many are still under
the impression that DATA_MODE remained a single-value variable.
So I'd like to raise this issue here: it is correct that some variables (such as
DATA_MODE) should have a N_PROF dim, but it is also important that some variables remain singlevalued. Of particular importance are JULD, LATITUDE, LONGITUDE, because the single-cycle files
absolutely need to convey the information that all primary and secondary measurements are
collected at the same time and location, even though they have different vertical sampling schemes.
Here are the variables that I think should NOT have the N_PROF dimension in the single-cycle files:
PLATFORM_NUMBER
STATION_PARAMETERS
CYCLE_NUMBER
JULD, JULD_QC, JULD_LOCATION
LATITUDE, LONGITUDE, POSITION_QC, POSITION_SYSTEM CONFIG_MISSION_NUMBER
KO, the single cycle files – and multi-cycle files have the same N_PROF dimension. This is not a new
feature. In the case of the single cycle files, N_PROF is usually equal to 1. The version 3.0 does not
change anything on that topic compared to previous versions.
From Claudia Schmid claudia.schmid@noaa.gov 20/05/2013 15:37
5903619_Dtraj.nc sounds good. The same scheme could work for meta and tech files if the need
arises.
Adding an R to the real-time files or not? Maybe it is better to do it, because then the users will find
out much quicker that we now have R and D traj (meta/tech) files.
OK, I update chapter 4.1 “File naming convention”
<FloatID>_<R/D>traj.nc
Examples
1900045_Rtraj.nc : real-time trajectory from float 1900045
1900045_Dtraj.nc : delayed-mode trajectory from float 1900045
Merging R and D traj files would be nice, but could be problematic for some floats. At one point we
had a traj file checker and merger, but the format changes plus the priority of DM-qcing profiles
made this program outdated a long time ago (again - resources are a challenge, in this case it was so
for both the DM operators and the DAC). How much effort should be put into this is not clear,
because once a float died and the DM-qc is completed, then the R file will be gone. If we do go for
the merging: Because such a program would work the same way across the board for all float types
(hopefully), it might be possible for one DAC to volunteer to develop a program that does the
merging?
The thoughts in the last paragraph are mostly also valid for the tech and meta files.
Two statements were not quite clear to me:
"I'm not a fan of having two files with non-overlapping data ..."
if we keep creating real-time traj files in the same way as now, then the R and D files will have an
overlap.
"Whenever we have two files with different quality data in different places ..." - why in different
places? The should be in the same place (the dac/wmoid directory).
Bye
Claudia
On 05/19/2013 09:50 PM, Ann.Thresher@csiro.au wrote:
> I am not sure this won't create confusion - we already have files with 'D' in the 'last' position for the
descending profiles. I like consistency so that would mean a D at the beginning - however... we
could put it either at the end of the number (5903619D_traj.nc) or before the file type
(5903619_Dtraj.nc) to avoid conflicting with the descending profile files.
>
> Another point - this may make it harder for the users - do we want to change the names of the
current files so they have R in them? or leave them as-is so users don't need to reprogram to read
them? Then only the D files would be identified as different.
>
> I'm not a fan of having two files with non-overlapping data though for traj files this appears to be
necessary. If possible, it would be good if we could actually incorporate the D moded data for tech
and metadata into a single file with the R data even if it means extra coding for the DACs. And this
eliminates the need for a different name - we just need a data mode variable to identify the stage of
a variable or cycle in the tech and meta files. Whenever we have two files with different quality data
in different places, we make it almost impossible for the users to find the best version without a lot
of extra work. Philosophically, I would rather do that work myself and serve a single, best version.
>
> Whatever we do should be effective in the long term.
>
From Annie Wong <awong@ocean.washington.edu> , 20/05/2013 23:39
Just spent some more time going through previous versions of the users manual.
I agree keeping N_PROF in the variables in Section 2.2.4 is best for the purpose of merging the
multiprof files. Thank you for pointing that out.
For JULD, LAT, LONG in particular, John and I both agreed that they ought to retain the N_PROF dim
to allow for future evolution of complex sampling in a single cycle. But we really (really!) need to
clarify this in Section 2.2.
I have some suggestions here - see attached. Hope this helps.
OK, I reported your suggestions in chapter 2.2 “Profile format version 3.0”
I am ambivalent about the proposal to add yet another variable, such as PROFILE_NUMBER as you
suggested. I can see how that can be useful for bounce profiles, but for most cases I can also see it
being a source of confusion. Perhaps other people can weigh in on this.
PROFILE_NUMBER will not be present in version 3.01, the decision to include it or not is delayed.
Thank you for Version 3.01. Please note that CONFIG_MISSION_NUMBER in the prof files has dim
N_PROF, not N_MISSION
OK, done
Zdzislaw Marek Stawarz Marek.Stawarz@bsh.de, 13/06/2013 10:15
I have a question about the data format. Some D-files I’ve recently submitted were rejected by the
format checker. One of detected errors concerns the variable FIRMWARE_VERSION:
>>
>>
>>
attributes: FIRMWARE_VERSION:conventions: Definitions differ.
conventions = "" ;
Data File = '" "'
It means, the format checker expects a <0x0 char> string. It corresponds to the Argo User’s Manual,
where the attribute conventions is set to "" and not to " " (blank) but I think it should be a typo ... It
does not seem to make sense to define an attribute with <0x0 char> … moreover it is a single
attribute with such convention throughout the manual … but maybe I overlook something.
Anyhow, the software used to create RT netcdf files enables to put such attribute value and the
checker expects it too.
My problem is, if I generate “own” netcdf files (sometimes I had to extract the single profiles from
multiprofile files) using MATLAB and netcdf toolbox, as well as MATLAB netcdf routines, I get an error
message: attribute value cannot be []. It is no problem to create an <0x0 char> MATLAB variable (e.g.
simple using char([]) but I cannot create a corresponding netcdf attribute value. I have tried different
Provided that there is no typo in the manual (in that case the checker should be modified and the
firmware_version:conventions=”” : this is a mistake, there is no particular convention for
firmware_version values, so this “conventions” attribute has to be removed from the manual
Frost, Mr. Michael Michael.Frost@nrlmry.navy.mil, 18/06/2013 18:03
I have created a new 3.0 spec file that follows the 3.01 spec that Thierry has provide in the .docx.
In doing so, I have made a few minor modifications to the doc, which I am providing back. These
were strictly syntax type issues. For example, the doc at times had a space between the colon and
long_name, which caused the format checker some issue. At times, the long_names had a space
before the end quote.
I reported your modifications in the manual.
John Gilson jegilson@gmail.com
13/06/2013 00:46
Yes the issue would have existed previous if anyone had decided to fill in the resolution with
anything other than 0.001. It's possible that the resolution for most float types is 0.001 for PSAL and
TEMP, however for the SOLO it is not. I'm spending a great deal of time making the new V3.0 files for
SIO and I'm trying to get everything as correct as possible. John
I updated reference table 3: For each parameter, the resolution attribute is mandatory. However,
the resolution value is sensor dependant.
Annie Wong awong@ocean.washington.edu
19/06/2013 21:36
Hi Thierry, thank you for incorporating the various comments for the users manual. But I think you
missed some comments from Esmee which she sent out in an email dated 7 June 2013.
I have attached a copy of her email here.
An important point in her email is that in the prof, traj and meta files, there should be two variables:
FLOAT_SERIAL_NO and PLATFORM_TYPE.
These two variables are used to replace the old variable INST_REFERENCE.
They have been recorded correctly in Table 2.4.4 for the meta files, but PLATFORM_TYPE is missing in
the prof and traj files.
In Table 2.2.4 for the prof files, please record these two variables:
FLOAT_SERIAL_NO(N_PROF, STRING16)
PLATFORM_TYPE(N_PROF, STRING32)
In chapter 2.2.4, I added the PLATFORM_TYPE variable and added an N_PROF dimension in
FLOAT_SERIAL_NO.
In Table 2.3.4 for the traj files, please record these two variables:
FLOAT_SERIAL_NO(STRING16)
PLATFORM_TYPE(STRING32)
In chapter 2.3.4, I added the PLATFORM_TYPE variable. The FLOAT_SERIAL_NO was already
present.
Megan Scanderbeg mscanderbeg@ucsd.edu
20/06/2013 21:02
Thanks for the updated version of the user's manual. I just looked at the trajectory section, including
reference tables related to the trajectory format, carefully and only found a couple minor corrections
which I made and have attached to this email.
I reported 2 corrections and ignored one.
1. Add a global attribute in the profile file :
:comment_on_resolution = "PRES variable resolution depends on measurement code";
2. Remove the STRING2 dimension from SATELLITE_NAME
char SATELLITE_NAME(N_MEASUREMENT, STRING2);
3. Remove the old reference table 15 (but keep the new one)
Finally, it is a great idea to provide links to draft versions of documents on the ADMT website. While
developing documents, it is great to have this place for all to access it. I am happy you have added a
link to the traj3.0 description on the "Draft documents" page. Once the
traj3.0 format is correctly described in the 3.01 version of the User's Manual, we don't need a link to
the draft one any longer.
Mike Frost Michael.Frost@nrlmry.navy.mil
24/06/2013 17:43
- Need to fix up this string in the manual: CONFIG_MISSION_NUMBER:long_name = "Unique
number denoting the missions performed by the floatMission";
Here is the new profile CONFIG_MISSION_NUMBER long_name:
CONFIG_MISSION_NUMBER:long_name = "Float's mission number for each profile";
Esmée Vanwijk Esmee.Vanwijk@csiro.au 27/06/2013 03:17
Annie - thanks for picking up on that.
I didn't have the original diagram but have amended the one in the manual with a bit of cut and
paste, attached here.
Hopefully that will do!
Thierry - could you please replace the diagram on page 49 with this one?
I've just edited it so it now refers to CONFIG_MISSION_NUMBER rather than the outdated
configuration_phase_number.
John Gilson jegilson@gmail.com
13/06/2013 00:46
Only one comment...by adding the PLATFORM_TYPE as a string32 to the trajectory file....the file now
requires the addition of a STRING32 definition.
There was no STRING32 dimensioned variable before PLATFORM_TYPE.
I immediately add this missing dimension.