Look into supporting higher dimensional data

[maximedion2]

The current reader is limited to reading1D, 2D and 3D data. I wonder if I could make this more generic and support arbitrary numbers of dimensions.

[jayendra13]

How about this approach https://gist.github.com/jayendra13/af793b3d377c4fa7d14147df7c2276df ?
ref:

• https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor
• https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor

cc: @alxmrs

[maximedion2]

How about this approach https://gist.github.com/jayendra13/af793b3d377c4fa7d14147df7c2276df ? ref:

* https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor

* https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor

cc: @alxmrs

hey @jayendra13! Thanks for the comment!

So I should clarify what this issue is about. The code in this crate directly reads the raw data from the chunks (actual zarr chunks, icechunk, etc...), so the dimensionality doesn't really matter for that, the data is just a sequence of bytes that gets decoded into something that is already "columnar" in a sense. The only reason there's a dependency on the dimensions is that I added support for "broadcasting" 1D to ND. Basically if you have some ND data, and N coordinates, my goal is that each coordinate can be stored and read as 1D data, and then broadcasted to ND (loosely speaking; it's not a real broadcast, it's more like extending the 1D data to have the right length) when needed. And that's only supported for N <= 3 right now. I'm hoping I can come up with a general approach that will support arbitrary numbers of dimensions, that's why I made this issue.