WASIp3, threading ABI changes, LLVM releases, and schedules

[alexcrichton]

WASIp3 is nearing publication, and with it async support in the component model. This is expected to be manifested in a new target, wasm32-wasip3, alongside existing targets. This is also expected to be manfiested into other ecosystems, such as Rust, as well. Spec-level documentation for the component model is in the explainer and marked with 🔀.

Another related component-model change coming down the pike "soon" is support for cooperative threading in the component model. This is annoted in the spec with the 🧵 emoji. The implementation of cooperative threading is not as far along as the implementation of async support + WASIp3 APIs at this current time, but it is well under way.

A major change coming with 🧵 is that the ABI of functions is going to change in two ways:

• The in-wasm shadow stack pointer will no longer be stored in a global, it's in context.get 0
• A thread's TLS address is not stored in a global, it's in context.get 1

These are mostly invisible changes to higher-level users but major changes from an ABI level. These changes require the component-model-async proposal as well as 🧵 and thus are expected to be exclusively scoped to the wasm32-wasip3 target. Preexisting targets, such as wasm32-wasip1, wasm32-wasip2, and wasm32-wasip1-threads are all going to be unaffected.

I'm going to set aside for now the rationale for these changes insofar as that's more of a technical discussion to have elsewhere (in theory). Instead what I'd like to focus on in this issue is the scheduling of releases for all of this. Given where the development of all this is, my personal understanding is:

• Implementation, specification, and support for WASIp3 and component-model-async are "nearly done".
• Implementation, specification, and support for 🧵 is underway, but still much more experimental than async support.
• Some folks expect WASIp3+async to be released first with 🧵 layered on afterwards. Some folks expect the opposite.

My personal rough estimation is that if 🧵 were to block a wasm32-wasip3 target then it would be on the order of ~6-12 months between WASIp3+async being "done" and shipping a usable target. WASIp3 and async are usable-ish on a wasm32-wasip2 target (that's how all development has gone so far), but it's easy to get things mixed up and have wires crossed accidentally. Put another way, power users could get things done, but it's an uphill battle trying to exclusively use preexisting targets for WASIp3 + async.

---

Given all this, this is where I wanted to outline my personal thoughts for how to thread these needles. Specifically what I'm imaginging is:

1. In the near future, WASIp3 is published. At this time component-model-async is considered finalized and cannot have breaking changes.
2. Shortly thereafter, the wasm32-wasip3 target is added to wasi-sdk and published here.
3. Shortly thereafter, the wasm32-wasip3 target is added to rust-lang/rust and published there.
4. Embedders, toolchains, libraries, etc, would start adding support, as appropriate, for wasm32-wasip3 and support would percolate throughout the ecosystem.
5. Development of 🧵 continues, using wasm32-wasip3 as a target. This will culminate in patches to LLVM, wasi-libc, and Rust for example.
6. Eventually an LLVM release will be made with 🧵 support. My expectation is that, at this time, the wasm32-wasip3 target unconditionally uses the new ABI and has no option to go back to using the old ABI. The old ABI is still used on wasm32-wasip2-and-prior.
7. This LLVM release is then pulled into wasi-sdk and a version of wasi-sdk is published. Rust updates quickly thereafter with this update.
8. Artifacts produced before this LLVM release for wasm32-wasip3 are just incomptatible with artifacts produced after the LLVM release. Probably some sort of wasm-ld-level enforcement with some sort of ABI version

The rough assumption is that (1) and (6) are far enough apart that there's a lot of value to be had in not delaying (1). The other rough assumption is that cross-pollination of artifacts from different versions of LLVM is, as of today, a somewhat niche case which is not rampant and a stable of every project. This in theory uniquely positions this to be able to be pulled off without much disruption. Inevitably some users will cross-pollinate LLVM versions but the theory is that this will be rare enough and something that can be easily rectified if it happens.

---

So that's all my thoughts on this. I want to cc a number of interested parties here as well to ensure y'all are also all on board with this as well. I also don't want to pretend that any of this is set in stone, if others have other thoughts/ideas please share and we should discuss!

cc @cpetig, @dicej, @lukewagner, @TartanLlama, @tschneidereit

[TartanLlama]

Yeah this all matches my expectations

[dicej]

Thanks for writing this up, Alex.

There are no ABI compatibility guarantees across major WASI-SDK releases anyway, so prominent users of dynamic linking (such as the Python ecosystem) should be prepared for this kind of thing already.

[alexcrichton]

That's true yeah that there's no guaranteed ABI stability, but so far wasi-sdk has practically been quite ABI-stable so I could imagine mixing artifacts across versions still comes up, if only by accident. An example of this is wasm-ld in rust-lang/rust. The version that rust-lang/rust ships is using whatever LLVM version rustc is using, but this is not tied to the exact same version that wasi-sdk used in CI (and used to ship libc.a with thewasm32-wasipN targets) is built with. Minor discrepancies like that still come up and are generally expected to work. That's where for rust-lang/rust we'll need to carefully coordinate the update there w.r.t. and LLVM release and a wasm32-wasip3 target, but those are details I'm confident we can handle.

[cpetig]

Thanks for writing this down in this detail, Alex!

As a bleeding edge user of the LLVM patches which enable 🧵 support I was hoping that (1) and (6) would be some days away from each other. Thus, I asked for avoiding a wasip3 release which doesn't already use the new context.get 0 for stack pointer. For wasi-sdk this should be possible as including a development version of LLVM containing the change would be possible, though somewhat more risky than only using upstream releases.

But I can understand that for the Rust project relying on patched pre-releases of LLVM is not an option. And making an exception for a single tier3 target sounds outrageous. Delaying the LLVM 22 transition in wasi-sdk for the other targets to wait for p3 readiness feels also wrong.

In an ideal world any compiler targeting wasip3 and referencing the no-longer-pre-release-dated WASI interfaces would already use context.get 0 from the get go. This way, there might be evolving thread primitives in the libraries, but at least the object files would be compatible across any wasip3 toolchain.

My feeling is that the LLVM changes for context.get are unchanging in the upcoming months and only the libraries would evolve. So there might be a chance of including these at an early stage when the full 🧵 support is not yet finalized. (Given the history of wasi32-unknown-unknown being incompatible between Rust and C for years, I really believe any quick decisions would be a mistake. Even if it would work out my way here.)

Having a Rust release which is binary incompatible to a C SDK release, because one side transitioned to context.get first, sounds like a bad user experience for wasip3. But I really don't know whether it is possible to avoid this situation.

[alexcrichton]

If a Rust target of wasm32-wasip3 ships before the context.get changes, then it's definitely guaranteed that there'll be a period of time where the latest wasi-sdk and the latest Rust stable differ. This is just due to the nature of Rust's release trains and how Rust needs to bundle wasi-sdk artifacts, so wasi-sdk needs to be released first.

I'm bullish on being able to pull this off myself. A key part of the problem with wasm32-unknown-unknown in Rust was that an existing user (wasm-bindgen) heavily relied on the "broken" ABI and didn't have a transition to the "correct" ABI. That ingredient is missing here as mixing C/Rust with different versions of LLVM will be broken and difficult to rely on the broken-ness.

I'll definitely acknowledge though that what exactly happens here will be highly dependent on the precise timelines. If we don't release WASIp3 til October then that changes a lot of this. If wasi-libc's WASIp3 port isn't ready to October that also changes things. Given that we can also reconvene here right before "pulling the trigger" of sorts and publishing a wasi-sdk with wasm32-wasip3 as a target. Doing such implies WASIp3 is released (spec-wise) and wasi-libc is ready implementation-wise. If LLVM-with-context.get is mere weeks away it seems reasonable to delay, but if we're looking at 6-12 months then that's a different situation (IMO)

[sbc100]

So is the plan to have a separate LLVM mode for "context.get" vs "global.get"? Is there a patch for this already? I think it might be good idea to align with shared everything threads if this is the plan since shared everything threads also need to move away from using global.get for SP & TLS base. (CC @tlively )

I guess its also a fairly easy transform to convert a binary from one standard to the other using something like a binaryen pass?

[alexcrichton]

I believe the work-in-progress PR is at llvm/llvm-project#175800. My impression is that this strategy loosely corresponds with the predicted direction of shared-everything-threads where there'll be something dedicated to the stack pointer which isn't global.{get,set}. If new wasm opcodes come around we'd likely define context.{get,set} in the component model to be exactly those opcodes and otherwise if it's "host magic" then things are already set up to take advantage of that.

And yeah it should be relatively simple to binary translate between various options. We already plan on doing so for the web embedding of components where global.{get,set} is likely to be sufficient and the JS host can manage as appropriate.

[tlively]

There is no consensus on how stack pointers will ultimately work with shared-everything threads and there is no path to achieving consensus until prototype implementations have made significantly more progress. The expected timescale for that is significantly longer than anything else mentioned in this thread.

But my understanding is that the component model folks have designed context.get with this uncertainty in mind and are comfortable moving forward ahead of shared-everything threads. I just want to make sure everyone is on the same page about how uncertain this is.