ODD PI 26.? Objective ?: WASM chunk codec research and PoC

[jkeifer]

Motivation

The vision behind this work is perhaps summarized with the goal of making data processing work like the web: portable, sandboxed, and trustless.

This PoC aims to prove that chunk decoding can be unified across chunked data format using a codec VM architecture. Just as web browsers execute untrusted code safely via sandboxing, datasets will be able to specify their own decoders (including novel or proprietary codecs) that consumers can execute without installation, compatibility concerns, or trust barriers. This foundational shift enables:

• Cross-format tooling that works universally: as Zarr has worked to unify array formats, we can go a step further and support formats beyond arrays
• Frictionless sharing of experimental/proprietary encodings: as data volumes exponentially increase, ensuring we can design and seamlessly use more optimal compression techniques becomes an imperative
• The prerequisite infrastructure for format-agnostic access protocols like CCRP

Description

For this initial PoC we aim to build a proof-of-concept codec virtual machine (VM) that can decode chunks from multiple data formats using a unified interface. Specifically, we have identified the following deliverables:

• Inventory of common codecs across common formats (Zarr, COG, TIFF, HDF5, Parquet)
• Basic open source codec runner with hybrid execution model (native + WebAssembly VM)
• Tests showing successful decoding of real array chunks
• A definition of the minimal chunk metadata content required to drive this initial codec executor
• An ADR-style document detailing the vision for this new codec ecosystem

Acceptance Criteria

We have each of the above deliverables and are ready to start engaging the rest of the ODD team and other community stakeholders in the discussion around the future of this work.

Sub-tasks

• Create codec inventory
• Create WASM codec implementation(s)
• Create WASM codec runner with tests
• Create demo(s) showing how this all works
• Write ADR outlining the ecosystem vision

[jkeifer]

The codec runner we've named chonkle and the repo is here. We have an initial demo notebook here. We'll see how this notebook evolves over the next couple weeks, or if we end up adding additional demo notebooks to show off new features.

The codec inventory currently exists as a first draft. It is entirely drafted by an LLM right now; it's unlikely that the full content will be validated anytime soon, but that's okay I think, as the intent is for it to be a guide for future work, not an authoritative reference. But I still expect revisions as more data formats are researched and a thorough read-through unveils inevitable hallucinations or other such errors.

We currently have a single codec, TIFF predictor 2, implemented twice: once in C and again in python. We want to show multiple patterns for how codecs can be developed for this system, hence the same thing in two languages. We might try to add a rust implementation, or we might be able to get to other codecs to show greater functionality.

[maxrjones]

FYI @jkeifer I'm going to change the label on this to "pi-objective-proposal" to match the issue title and align with the objective review process

[jkeifer]

We've wrapped up all the work on this objective given the time box we were allotted. We had five deliverables, the status of which is included below.

Create codec inventory

This is has been merged into the spec docs outlined below. The entry point into this inventory is specifically in this document. This was entirely generated by an LLM with prompt iteration; it is not expected to be 100% accurate nor fully detailed, but to be a reference for helping to understand what codecs need to be implemented and how to identify similarities across existing implementations to help unify codecs where possible. It's expected that formal specification of codecs would ultimately replace this inventory.

Create WASM codec implementation(s)

We implemented several codecs. Most are embedded in the chonkle repo at the moment, to simplify iteration. Those include:

• identity-c (WASM component model)
• tiff-predictor-2-c (WASM component model)
• zlib-rs (WASM component model)
• identity-core-c (WASM core ABI)

We also have two standalone repos with the following codec implementations:

• tiff-predictor-2-python (WASM component model)
• tiff-predictor-2-c (WASM core ABI)

The standalone codec repos were valuable in validating remote storage and fetching of WASM modules.

We also developed a way to leverage numcodecs codecs as "native" codecs.

Create WASM codec runner with tests

This is chonkle. It is a python proof-of-concept/research prototype we built to vet this idea and show that it is functional. Chonkle is the tooling that enables the following demo deliverable.

Create demo(s) showing how this all works

We have a demo notebook within the chonkle repo that shows how all this works.

Write ADR outlining the ecosystem vision

This ended up being initial spec documentation for the ecosystem, not really an ADR. Some of it is more thoroughly considered than others--we've attempted to catalog all open questions and spec areas obviously needing more work within the documentation--but it's there, and it presents a fairly comprehensive picture of the immediate vision. All of this is contained within the cylf protospec repo.

Turns out that we were so successful with chonkle as a prototype implementation that we've already identified many spec improvements from the lessons we learned implementing. Those improvements have been folded into the spec where possible, so the spec is now leading the implementation.