[Bug] `get_keepbits` returning a negative integer value for `.grib` datasets

[fnhirwa]

• xbitinfo version: 0.1.dev587+g10a08dd
• Python version: 3.9.18
• Operating System: `macOS Sonoma Version 14.4 (23E214) (Apple M1 Processor)

Description

Was trying the xbitinfo and following the tutorial presented and when compressing the .grib datasets it seems that the compression fails as the get_keepbits function returns negative values.

What I Did

import xbitinfo as xb
import xarray as xr

dataset = xr.tutorial.load_dataset("era5-2mt-2019-03-uk.grib").astype("float32")
info_p_bit = xb.get_bitinformation(dataset, dim="longitude", implementation="python")
keepbits = xb.get_keepbits(
    info_p_bit, 0.99
)  # returns negative number due to lossy compression
ds_bitrounded = xb.xr_bitround(
    dataset, keepbits
)  # raises a warning as keepbits is negative

[sarthakvyas16]

i want to fix this issue please assign me this issue.

[observingClouds]

@sarthakvyas16 before assigning you to the issue it would be good to hear your ideas and discuss them.

[sarthakvyas16]

ok let me explain.
First of all i m review the implementation of get_keepbits. Understand its expected input, output, and the logic that leads to the computation of the return value after that Check the input being fed into get_keepbits. Ensure that the .grib datasets are formatted correctly and do not contain any corrupted or unexpected values and then ensure that all variables used in the calculations are of the correct data type. Sometimes, implicit type conversions can lead to unexpected negative values and tracing the calculation which help us to identify where the negative entity is originating.
and many more things which use to sortout bug.If you are satisfy by my Point of view , then assign me this issue.

[jweisbaum]

I am also encountering this, but I don't understand why grib compression would be the cause. I tested this with both jpeg compressed and simple packing gribs and got negative keepbits, but I don't know that the grib compression is definitively the cause. If I do a ds = xr.load_dataset('blah.grib', engine='cfgrib'), I would expect that the data is not lazily loaded, but fully loaded, and therefore the values of the ds would be decompressed, and thus, return a positive number. Can we get some confirmation that 1) the negative keepbit numbers are in fact a result of grib compression, and 2) what a reasonable workaround would be, either from how we load and format xr datasets or via xbitinfo? Thanks.

[miniufo]

I also encountered this, but with my own dataset:

import xarray as xr
import xbitinfo as xb

ds = xr.open_dataset('/Data/2DTurbulence/4096/Output2D_4096_6.nc', chunks={'time':1}).isel(time=-5).load()
print(ds)
print('\n')
bitinfo = xb.get_bitinformation(ds, dim='x')  # calling bitinformation.jl.bitinformation
keepbits = xb.get_keepbits(bitinfo, inflevel=0.99)  # get number of mantissa bits to keep for 99% real information
print(keepbits)

with the output:

<xarray.Dataset>
Dimensions:  (x: 4096, y: 4096)
Coordinates:
  * x        (x) float32 -3.142 -3.14 -3.139 -3.137 ... 3.135 3.137 3.139 3.14
  * y        (y) float32 -3.142 -3.14 -3.139 -3.137 ... 3.135 3.137 3.139 3.14
    time     float32 1.992
Data variables:
    u        (y, x) float32 2.185 2.186 2.186 2.186 ... 2.194 2.193 2.19 2.188
    v        (y, x) float32 -1.259 -1.254 -1.25 -1.246 ... -1.271 -1.269 -1.264
    vor      (y, x) float32 5.764 5.166 2.659 5.911 ... -1.822 2.282 4.818


Processing var: vor for dim: x: 100%              3/3 [00:04<00:00,  1.55s/it]
<xarray.Dataset>
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 'x'
  * inflevel  (inflevel) float64 0.99
Data variables:
    u         (inflevel) int64 7
    v         (inflevel) int64 6
    vor       (inflevel) int64 -1

It is a bit strange that the vor is -1, which cause later ValueError: keepbits must be zero or positive. The vor variable looks OK by plotting its distribution. So why does it get a -1?