Serializers for ndarrays

[maartenbreddels]

cc @jasongrout @vidartf @SylvainCorlay
I want to transfer some ndarray from python to js (python side numpy, js side scijs/ndarray). It would be nice to have a js package for this, as well as a place to put the python code.

The python code would make sense to put here I think, but also the js code (fan of monorepos!).

We (/me, @vidartf and @SylvainCorlay) also discussed using arrow to transfer tensors, which would make it easier for kernel backends in for instance c++ to handle more complex tensors.

However, I think simply transferring a memory buffer, the shape and dtype (maybe strides, but C_CONTIGUOUS is fine with me first) will handle >80% of the cases.

So is this repo the place to put a js package (name suggesions welcome, jupyter-ndarray-serialize?)

[vidartf]

Currently, I've done some work in ipydatawidgets, which includes basic serialization to scijs ndarrays. It also includes separate data widgets for sharing data. There are still some TODOs left, and I've been doing some prototyping on more advanced data widgets there as well, but the basic functionality should be reasonably solid.

Note: I'm currently refactoring quite a bit of the code to ensure that projects like pythreejs can use it without needing the NB extension to be installed (required for data widgets, but not direct serializers).

I'm more than happy to break up / merge repos if anybody has a suggestion for a better way to organize things, but for now I needed a repo where I could quickly iterate on ideas and concepts.

[vidartf]

@maartenbreddels / @SylvainCorlay Do you think I should add serializers in ipydatawidgets for serializing to/from something else than scijs ndarrays? Adding another set of serializers should be no problem!

[maartenbreddels]

Maybe to just typed arrays, for situations where you just want 1d arrays.

[ianhi]

What is the status of this? I started using ipydatawidgets and it's great. Maybe as an intermediate step the part of the readme that reads:

Another goal is to create adequate serialization and deserialization routines for these trait types to be used with the ipywidgets project (to_json and from_json). These could also return a list of binary buffers as allowed by the current messaging protocol.

Could also link to ipydatawidgets?