Implement lossy compression via quantization

[davidhassell]

Lossy compression via quantization was introduced in CF-1.12 (section 8.4). We need to implement this in cfdm.

Requirements

1. Read quantization metadata from a dataset, store it in memory, and write it back out to disk

2. Allow quantization metadata stored in memory to be modified.

   • But this should not be too easy, because changing the metadata will not change the actual quantization of the data. The use case is correcting mistakes in the quantization metadata.

3. Quantize data during the process of writing to disk, creating the correct quantization metadata in the output dataset.

   • But only if the data are not already quantized, as determined by the absense of quantization metadata. We should not allow the quantization of quantized data.

Proposal

• A new cfdm.Quantization class to store quantization metadata.
• New methods to get/set/del quantization metadata:
  • get_quantization, _set_quantization, _del_quantization. The set/del methods are underscored to help warn that their use will not result in a change in the data, and may result in the quantization becoming inconsistent with the data.
• New methods to get/set/del an instruction to quantize the data when writing it to disk with cfdm.write. These methods do nothing until the time of writing:
  • get_quantize_on_write, set_quantize_on_write, del_quantize_on_write. These methods do not need to be underscored, since there is no risk of inconsistencies.
  • If quantization metadata already exists then set_quantize_on_write will fail.
  • If a quantize-on-write instruction already exists then _set_quantization will fail.
• The actual quantization of data is handled trivially (from our point of view) by the netCDF4 Python library.
  • I notice that netCDF4 Python can not yet handle the CF-allowed DigitRound quantization. I presume that this is because netCDF-C doesn't handle it, but I haven't checked on this.

PR to follow imminently.

(Pinging @czender, just so you're aware of other implementations out there)

[czender]

Hi @davidhassell Best wishes with this feature. You are right that DigitRound is allowed by CF and is not (yet) in the netCDF library. There were permissions issues, grant deliverables, and social dynamics that determined which algorithms went into netCDF (and what they were named). DigitRound ended up being left out though someone might add it in the future. My understanding is that the netCDF4 Python library uses the quantization from the netCDF C-library, so it's not there either.

[davidhassell]

Thanks for the background, @czender. I suppose I could implement DigitRound here in cfdm, but I don't really want to! I wonder if anyone in CMIP7 will want to use it - that might provide some impetus for getting it into netCDF?

[czender]

Granular BitGroom uses the DigitRound method to determine the number of bits to quantize in a number. In that sense, DigitRound is in netCDF. Granular BitGroom extends the DigitRound algorithm by using IEEE rounding and handling missing values and NaNs the way CMIP7 needs them to be handled (i.e., do not quantize them). It would still be good to have DR more readily accessible so it could be entered in a "bake-off" of all the quantization algorithms.