File Publication Through GTS-to-WIS2 Gateway

[masato-f29]

The approved publication "Provisions for the Transition from WIS 1.0 and GTS to WIS 2.0" states that the GTS-to-WIS2 gateway will support the topic hierarchy using the GTS abbreviated heading:
origin/a/wis2/{centre-id}/data/[core|recommended]/T1/T2/A1/A2/ii/CCCC
My concern as the GTS-to-WIS2 gateway operator is how to publish files following the naming convention specified in the GTS Manual, such as the example below:
W_au-BOM-mel,AMSUA,RARS+NOAA19+csy_C_AMMC_20241123040541_amsual1c_noaa19_20241123_0354_81408_bufr.bin
Should a new topic hierarchy be created for such files, or should these files be published under the current topic hierarchy of T1/T2/A1/A2/ii/CCCC?

[amilan17]

3.1.3 Technical requirements
A GTS to WIS2 gateway is a DCPC function. All requirements related to WIS2 Nodes are applicable. A GTS to WIS2 gateway will obtain a specific unique centre-id from the WMO Secretariat;

In addition to the standard WIS2 Node specifications, the GTS to WIS2 gateway will support the following:

The topic hierarchy for GTS data on WIS2 will be: origin/a/wis2/{centre-id}/data/[core|recommended]/T1/T2/A1/A2/ii/CCCC

Example for DWD: origin/a/wis2/de-dwd-gts-to-wis2/data/[core|recommended]/T1/T2/A1/A2/ii/CCCC

Example for JMA: origin/a/wis2/jp-jma-gts-to-wis2/data/[core|recommended]/T1/T2/A1/A2/ii/CCCC

The T1/T2/A1/A2/ii/CCCC above is derived from the headers of the data received on the GTS

[tomkralidis]

2024-11-26

• would these files have the AHL in the first line of the content? Can this be used to define the topic
• easy enough operation, but would be expensive (open/close file, read)
• @masato-f29 to continue discussion w/ DWD
• @josusky any comments here would be valuable

[josusky]

@tomkralidis I have subscribed to origin/a/wis2/de-dwd-gts-to-wis2/data/# and so far I got only files with names like:
ISII01SABM271500. The data ID is wis2/de-dwd-gts-to-wis2/data/core/I/S/I/I/01/sabm/ISII01SABM271500 (that is probably not very relevant, but seems ugly to me so I am pointing it out :-)). The notification contains, in the properties section, except other:

     "gts" : {
         "cccc" : "sabm",
         "ttaaii" : "ISII01"
      },

(Again it is ugly that the value of cccc is lower case and ttaaii is upper case, just saying.)
and the file itself starts like this (first two lines of the hexdump follows):

00000000  49 53 49 49 30 31 20 53  41 42 4d 20 32 37 31 35  |ISII01 SABM 2715|
00000010  30 30 0d 0d 0a 42 55 46  52 00 43 8d 04 00 00 16  |00...BUFR.C.....|

in other words, I can see the (in)famous GTS abbreviated heading there in four places (if not more):

• In the topic.
• In the data_id (twice in fact, once in the "path" and once in the "filename").
• In the WNM properties.
• At the beginning of the file.

Having said this, I did not see any W_au-BOM-*_bufr.bin so I cannot say if those have the AHL elements (TTAAii CCCC) somewhere or not. Perhaps @masato-f29 can provide more information about them.

[kaiwirt]

Maybe one could use the same topic as one would for data on WIS2. That way we would not need to invent something different. So data that is forwarded from the gtstowis2 gateways either comes in the TTAAii topic hierarchy or if that data does not have a TTAAii header in the standard wis2 topic hierarchy (which still can be sorted out by the centre-id containing gts-to-wis2)

[masato-f29]

Some file data, such as W_au-BOM-*_bufr.bin, include the AHL in the first line of their content, while others, like Z__C_RJTD_*_bufr4.bin, do not.
In addition to @kaiwirt's proposed solution, one potential approach could be to organize all GTS files into a simplified hierarchy, as shown below:
origin/a/wis2/jp-jma-gts-to-wis2/data/core/file
This structure could reduce the burden on gateway operators by removing the need to check the content.
If not convenient for data consumers, the structure could be further refined into a more detailed hierarchy aligned with the GTS naming convention.

[tomkralidis]

Bumping. @josusky or others involved in WIS2 Gateway testing: was this discussed or brought up as an issue?

[josusky]

The testing were I participated, as a consultant for UK Met Office, was for WIS2 to GTS gateway. This issue is related to the oposite gateways GTS to WIS2.
Those gateways currently use topics like this one:
origin/a/wis2/jp-jma-gts-to-wis2/data/core/I/E/I/X/07/EUMR
or
origin/a/wis2/de-dwd-gts-to-wis2/data/recommended/I/U/A/X/02/KARP
and also put the "gts" properties into notifications. I have not seen lowercase CCCC but I did not analyze to hard.
I would say that this issue was overcome by circumstances, but I let @kaiwirt and @masato-f29 to tell the last word.