icechunk-js twins!

[Shane98c]

Just wanted to note that it looks like we've been on more or less the same development track for this! https://github.com/carbonplan/icechunk/tree/icechunk-js/icechunk-js

Curious if you think we should consolidate efforts. I'm still waiting to hear more from the icechunk team on where they thing something like this should live. earth-mover/icechunk#1566

A quick glance shows that we're pretty close on functionality. It looks like yours has a nicer cache implementation and ours has support for icechunk v2.

Did you have thoughts on namespace? Saw you're not on npm yet #2

I was thinking of maybe putting something up at @carbonplan/icechunk-js so I could move forward with some zarr-layer work carbonplan/zarr-layer#21 edit: ended up pushing this up but happy to deprecate if we decide on a better home! https://www.npmjs.com/package/@carbonplan/icechunk-js

[espg]

Hi @Shane98c , I didn't have notifications set up on this repo, so just seeing it now. Yeah, I'm open to consolidating efforts, that sounds like it makes sense and I would love to get an implementation that combines functionality for both!

I actually am looking at putting this on npm shortly; we have a viewer that uses this + zarita as dependencies, and is callable via myst and quarto directives... in order to handle building the extension correctly, it makes sense to have an npm install path CI builds on the viewer directives.

As a random aside, you and I really, really do seem to have a thing for hitting all the namespace conflicts-- my first name is also Shane 😂

[mpiannucci]

Hey!! I am an icechunk maintainer from earthmover and I have been (finally) actively working on this. I have added our alpha node/wasm bindings here: earth-mover/icechunk#1630

You can also try it out on npm: https://www.npmjs.com/package/@earthmover/icechunk/v/2.0.0-alpha.5

This is early but im hopeful that we can come together to support this fully for the community. Happy to set up time to hcat about this if you all want to sync up together on it.

Because this is wasm it will need some custom work to get wasm storage backends built, but the node bindings will work as is in python icechunk.

[kylebarron]

@mpiannucci I think for the purposes of this discussion, we're more interested in the browser than in Node. So having a pure-TypeScript reader is a lot more appealing than WASM bindings, especially if they only support Node and server-side wasm

[mpiannucci]

It supports browser wasm, but needs work to include storage specific for use in the browser

But I also understand why many would prefer TS because of size constraints.

[kylebarron]

I think it depends on the complexity of the icechunk spec.

If the icechunk spec is so complex that read-only access is only feasible with bindings to the reference implementation, then sure, allocating resources into the wasm bindings makes sense.

But for a constrained but relatively high-value read-only environment like the browser, having a pure TypeScript implementation would be much preferred if the maintenance requirements of a read-only implementation are not high. Especially so because a wasm implementation would be no faster than a pure TS implementation (ignoring bundle size) because at the end of the day, you're just fetching a range request and interpreting it as an NDArray, no? No part of that would be faster in Wasm than in pure TS.

[mpiannucci]

If the icechunk spec is so complex that read-only access is only feasible with bindings to the reference implementation, then sure, allocating resources into the wasm bindings makes sense.

I dont think this is true today but soon we will be supporting two versions of the icechunk spec at all times and things become more complex.

No part of that would be faster in Wasm than in pure TS.

Agreed.

But as soon as someone who is not your group wants to use icechunk from node (i know other groups running js + zarr on the server), the rust core is so much faster than wasm can ever be. And we get the wasm bindings for free in lock step with the other icechunk bindings. So yes a lot of the reasoning is around maintenance.

[kylebarron]

icechunk from node (i know other groups running js + zarr on the server),

I'm not sure I understand your argument; if anything I feel like this is an argument in my favor? At least in the read-only case, even in Node a pure Typescript implementation would probably be at least as fast as Rust compiled to Wasm.

[mpiannucci]

I'm not sure I understand your argum

All im saying is that we are supporting node/bun/deno with napi-rs and with that we get wasm bindings at low maintenance cost (with some ability to optimize) so they will be available from our package.

[mpiannucci]

Oh i missed this:

At least in the read-only case, even in Node a pure Typescript implementation would probably be at least as fast as Rust compiled to Wasm

This is NOT true. We are compiling to WASM AND to native node. When running node you run with the rust core library, which should def be much faster than a pure TS implementation

[espg]

Sounds like a chat would probably be useful-- here's a when2meet for next week + this Friday , happy to set up zoom and send out invites for folks on the thread that are interested.

[espg]

Great, invites sent for Wednesday the 18th!

[lazarusA]

Just to summarise, there are currently these options:

• https://www.npmjs.com/package/@carbonplan/icechunk-js
• https://www.npmjs.com/package/@earthmover/icechunk/v/2.0.0-alpha.5 (no typescript)
• https://www.npmjs.com/package/@englacial/icechunk-js

[mpiannucci]

https://www.npmjs.com/package/@earthmover/icechunk/v/2.0.0-alpha.5 (no typescript)

Has typescript types

[mpiannucci]

Hi all, I had a personal conflict come up from 12-230 eastern on Wednesday, is there another time slot that may work for all?

[espg]

Looks like 3pm Eastern / Noon Pacific works...