Add support for MTG Satellite processing

[jacobbieker]

There is the new satellite from EUMETSAT for the 0 degree service that is now operational. It has a faster refresh (10min vs 15) and more channels (16 vs 12) and higher spatial resolution, so would be good to include.

Detailed Description

The MTG satellite is now fully operational, providing 16 spectral channels, with much higher resolution data than the MSG currently being used. The MTG will also eventually replace the MSG for the 0 degree service sometime next year (timeline is 6 months after it is operational).

Context

Higher resolution, more frequent data (every 10 minutes) can help with forecasting. It would also be nice to be able to have it in the GCP archive to make it more easily accessible.

Possible Implementation

It is already available in the data store from September to now, both the high resolution data, and the nominal resolution data, in the Data Store.
https://data.eumetsat.int/data/map/EO:EUM:DAT:0665
https://data.eumetsat.int/data/map/EO:EUM:DAT:0662

[jacobbieker]

Few updates here:

Once pydata/xarray#9948 is merged, we could then write sharded Zarrs, which should cut down the massive number of chunks on disk we have with the current satellite imagery (and would be worse with MTG)

Using xbitinfo to get how many bits we need to keep 99.9% of the information in the imagery, we could save half the storage by going from float32 to float16.

High resolution 1km
<xarray.Dataset> Size: 28B
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 4B 'x'
  * inflevel  (inflevel) float64 8B 0.999
Data variables:
    ir_105    (inflevel) int64 8B 9
    ir_38     (inflevel) int64 8B 8

High resolution 500m
<xarray.Dataset> Size: 28B
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 4B 'x'
  * inflevel  (inflevel) float64 8B 0.999
Data variables:
    nir_22    (inflevel) int64 8B 12
    vis_06    (inflevel) int64 8B 15

Low resolution 1km
<xarray.Dataset> Size: 76B
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 4B 'x'
  * inflevel  (inflevel) float64 8B 0.999
Data variables:
    nir_13    (inflevel) int64 8B 12
    nir_16    (inflevel) int64 8B 3
    nir_22    (inflevel) int64 8B 12
    vis_04    (inflevel) int64 8B 17
    vis_05    (inflevel) int64 8B 4
    vis_06    (inflevel) int64 8B 4
    vis_08    (inflevel) int64 8B 4
    vis_09    (inflevel) int64 8B 15

Low resolution 2km
<xarray.Dataset> Size: 76B
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 4B 'x'
  * inflevel  (inflevel) float64 8B 0.999
Data variables:
    ir_105    (inflevel) int64 8B 9
    ir_123    (inflevel) int64 8B 9
    ir_133    (inflevel) int64 8B 9
    ir_38     (inflevel) int64 8B 8
    ir_87     (inflevel) int64 8B 9
    ir_97     (inflevel) int64 8B 9
    wv_63     (inflevel) int64 8B 9
    wv_73     (inflevel) int64 8B 10

For these, we could either keep float32 and use BitRound to reduce storage costs by making it more compressible, or change to float16, and potentially also use BitRound. The one issue is the ones that are quite close to or above 16 bits of info, the high resolution 500m visible channel, and the vis_09 and vis_04 channels in the low resolution as well.

@devsjc Any thoughts on what might be better approach? Wouldn't want to get rid of info, could also keep only some of the variables in float32, and the rest in float16 I guess

[devsjc]

I think rounding off the data is a good idea, there's no need to hold extra precision. Are there any downsides to mixing float16 and float32 between data vars?

[jacobbieker]

No downside on mixing data types, they are stored completely separately, main question I guess would be if we can reduce all the data vars to float16 instead, and how much difference does it make for the imagery, so I'll keep looking into that and see what the difference is.

[jacobbieker]

So after some different tests on compression, making ocf_blosc2 support Zarr V3 and trying that, and using bitrounding to make the data easier to compress, here is the final suggested compression, etc. for the imagery:

float16, bitround to keep 9 bits, Blosc compression with Zstd level = 9, and only keep the highest resolution version of each channel

This results in about ~224GB per day of imagery, which is slightly less than the ~288 GB of NetCDF files for a given day.

This was the smallest on-disk storage from multiple tests, ocf_blosc2 resulted in higher on-disk storage interestingly enough.

Bitrounding does remove some precision, but only very little (dynamical.org does it for their archive, and they say it only changes values by at most 0.2%), and also fits in to the restrictions of not being able to redistribute the original values, as these values have been converted by Satpy, and reduced from float32 to float16, then rounded by 1 bit.

Now that pydata/xarray#9948 has been merged, we should be able to do this with sharded Zarrs as well.