Implement fixedscaleoffset codec

[kleinschmidt]

This implementation is my (i.e. a TypeScript non-knower's) attempt to pattern match the
other codecs here along with @manzt's suggestion in
manzt/numcodecs.js#49 (comment).

One thing I noticed is that the ArrayArrayCodec type has a single type
parameter (I think?). Does that imply that all array-array codecs must output
the same type that they accept as input? If so, I think that's probably too
restrictive; the python fixedscaleoffset codec explicitly supports encoding to a
different type than the input uses, and my intended use case is to decode
int16-quantized floats.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1211799105283496

[changeset-bot]

⚠️ No Changeset found

Latest commit: 6e0129b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[d-v-b]

zarr v3 data types can be a string or a JSON object with type {name: string, configuration: object}. See an example here

[kleinschmidt]

this one i think needs to be restricted to number types; can numeric types also have that format? is that how endianness is stored?

[kleinschmidt]

ah I see in the spec:

Each data type is associated with an identifier, which can be used in metadata documents to refer to the data type. For the data types defined in this specification, the identifier is a simple ASCII string. However, extensions may use any JSON value to identify a data type.

[kleinschmidt]

and endianness is specified as a bytes codec, I'm inferring.

[d-v-b]

yeah the bytes codec sets the endianness of the encoded data, but for decoded data, endianness is up to the implementation

[kleinschmidt]

testing on v3 has made me realize that coerce_dtype is really only meant for v2 data type strings, so this will need to be adjusted.

[kleinschmidt]

I think resolving the v2 vs. v3 issues may be trickier than I'm prepared to take on right now. Would you be willing to accept v2-only support @manzt ?

The numcodecs.zarr3 python functions generate a different ID (numcodecs.fixedscaleoffset vs. fixedscaleoffset in v2) so leaving this as-is will simply error.

[emmanuelmathot]

Very interested by this codec. Also for V3. Any progress on merging this one?
cc @ahocevar

[d-v-b]

the fixedscaleoffset codec should not be used for zarr v3 data because it does not define how values from one data type should be cast to values of another data type. the old zarr v2 fixedscaleoffset codec basically just used numpy's casting behavior, but that's not really workable in zarr v3 because zarr v3 has a formal data type model, and we don't want to be dependent on the runtime behavior of numpy for something that spans multiple programming languages.

thanks to a lot of community involvement we made a new codec in zarr-extensions called cast_value that is narrowly scoped to managing casting between ints and floats. the tl;dr is that the codec configuration contains a declaration of:

• the target data type
• a rounding mode
• a mode for handling out-of-range values
• an explicit input scalar : output scalar lookup table (necessary for non-numeric values like NaN)

The codec must halt if any value isn't covered by rounding, the out-of-range mode, or the lookup table. This, combined with the explicit lookup table, can be complicate performance in interpreted languages.

@manzt what effort is needed to get this into zarrita? the scale-offset part is really easy but the casting part might be warrant some performance considerations. I have a rust implementation here that could be compiled to wasm, but I don't know the rust + js interop story at all, so no clue if that work is worth the performance benefit.

casting is the tricky part of the scale-offset transformation. the actual scaling + offsetting is simple (but we also have a codec for that.

[d-v-b]

@kleinschmidt are you interested in working on a zarr v3 implementation of the scale offset functionality?

[manzt]

@manzt what effort is needed to get this into zarrita? the scale-offset part is really easy but the casting part might be warrant some performance considerations. I have a rust implementation here that could be compiled to wasm, but I don't know the rust + js interop story at all, so no clue if that work is worth the performance benefit.

I'd be up for adding a pure JS version in zarrita proper (at least initially). Folks can dynamically swap in codecs with zarr.registry.set, which I think satisfy the case where one wants a faster implementation (at the cost of a much larger bundle size with WASM).

[kleinschmidt]

@kleinschmidt are you interested in working on a zarr v3 implementation of the scale offset functionality?

It's not currently a priority for us at the moment so not for the foreseeable future.

[d-v-b]

sounds good, I'm happy to take this up