Hi! I have some model output I am trying to serve with xpublish (and xpublish-host and kerchunk). This model output has compression, and when I try to access it through the server I get the following error, though I am able to just open the model output fine (pointing to the kerchunked file):
TypeError: Object of type Shuffle is not JSON serializable
Indeed, a "Shuffle" object is in the metadata, for example here is what it shows in the xpublish code for an example variable:
'Uwind_eastward/.zarray': {'compressor': {'id': 'zlib', 'level': 3}, 'filters': [Shuffle(elementsize=4)], 'chunks': [1, 362, 794], 'dtype': '<f4', 'fill_value': 'NaN', 'order': 'C', 'shape': [1, 362, 794], 'zarr_format': 2}
I'm not sure why the compressor items remained a dictionary but the filter item was converted to an object, which is what seems to cause the issue. In the metadata coming out of kerchunk the filter item looked like
"filters":[{"elementsize":8,"id":"shuffle"}]
(I'm not sure if that change in "elementsize" is important as I go back and forth between two cases I'm testing, but I can track it down if it seems relevant.)
Does anyone follow this? I am not sure how to set up an actual working example for this workflow yet.
Hi! I have some model output I am trying to serve with
xpublish(andxpublish-hostandkerchunk). This model output has compression, and when I try to access it through the server I get the following error, though I am able to just open the model output fine (pointing to the kerchunked file):Indeed, a "Shuffle" object is in the metadata, for example here is what it shows in the
xpublishcode for an example variable:I'm not sure why the compressor items remained a dictionary but the filter item was converted to an object, which is what seems to cause the issue. In the metadata coming out of
kerchunkthe filter item looked like(I'm not sure if that change in "elementsize" is important as I go back and forth between two cases I'm testing, but I can track it down if it seems relevant.)
Does anyone follow this? I am not sure how to set up an actual working example for this workflow yet.