Separately owned handles for read/write ends of futures/streams

[alexcrichton]

I talked about this with some folks today but wanted to file an issue with our discussion and follow-ups. Today we realized that there is a previously-unknown issue with the semantics of "just a single handle" for both the read/write end of a future/stream within a single component, namely that if a polyglot scenario is considered where multiple languages are linked into the same component with shared-everything-linking then there's not necessarily a guarantee that the reader and writer are using the same canonical ABI. For example one might be using utf8 and one might be using utf16. With only having a single handle there's no easy way to arbitrate this across languages and solve this.

Personally I also have had concerns about having just a single handle for reading/writing as I feel like the resulting bindings/state machines that guests need to handle are quite complicated. One of the basic protections this is providing to the host, however, is that there's no need for a host to lift and lower to the same linear memory. That would make the side-effectful-order of lifts/lowers to become significant which is something we want to avoid specifying.

The basic idea for this issue is along the lines of:

• Creating a future/stream returns two handles: the reader end and the write end. For now the write end can't leave components but the reader end can.
• When a read/write are paired within the same component instance "somehow" the runtime returns this via a status code but doesn't actually do any lifting or lowering.

The hope is that the guest can easily handle this situation (somehow) and the host also doesn't have to do any lifts/lowers itself. In talking through this design space however we were unable to reach any firm conclusions about how exactly to slice/dice this. For example:

1. One possibility is to for the host to wait for the read/write to be paired. When this happens (regardless of which came in second) the read side completes first with a status code indicating "a local read completed". The {stream,future}.read intrinsic would probably take a third argument of "local result area" where the pointer passed to {stream,future}.write is written passed in. This measn the stream/future is now in a state where no future reads can complete until this read is "acknowledged". This acknowledgement would happen via a new component builtin. The reader would itself perform the lift within linear memory so it's purely the guest's responsibility here. Upon finishing the lift the reader would acknowledge the local read (probably from this "local return area") and then that would queue up a completion notification for the writer. The downsides here are:
   • This just specifies that the reader completes first. That means if the writer came second it's blocked to go to the reader and the reader acknowledges. Shouldn't the writer also be able to do the same thing?
   • More intrinsics/component builtins.
   • This "auxiliary return area" is somewhat nontrivial
   • This needs to generate a trap if the read/write intrinsics have differing canonical ABI options. Removing this trap is probably going to require "lazy lifting/lowering" in the future and won't be trivial. In the meantime it's not easy to get around the trap.
2. Another possibility is to trap the reads/writes and unconditionally (or fail them somehow) and require cooperation on behalf of the guest. It's a nice property today that if a T is sent along a future it requires no copies/lifts/lowers in the guest since it's known to be local, and ideally that could be preserved. This idea isn't fully fleshed out though.

Overall we reached the conclusion that to solve the original issue of polyglot components with heterogenous sets of options on lifts/lowers that we probably want a separate handle for each half of a future stream. How eactly to design this though is still uncertain.

[alexcrichton]

Ok here's a stab at a more fleshed out idea:

The overall idea is that language primitives always use canonical ABI intrinsics. Upon reading or writing if the "other end is available" (aka when reading there's a pending write or when writing there's a pending read) then that call immediately returns with a new status code indicating "the operation completed" with a LOCAL code or something like that. The read/write intrinsics take a new argument which is a pointer to a "local" location. This point has size/alignment fixed per intrinsic and basically is used to store the arguments of the other half. For example the future.read intrinsic would take a pointer-to-a-pointer where if LOCAL is returned the pointer-to-a-pointer is written to with the pointer argument to the future.write that the other half is doing.

Upon receiving LOCAL the language/bindings are responsible for using the "local area" to decode the result. In the case of future.read it would lift the value from the pointer given. In the case of future.write it would lower the value into the destination. In both situations after receiving LOCAL this would then be coupled with a new intrinsic: {future,stream}.complete-local. This would signal completion of the local operation (read or write) and would unblock the other end.

The "local area" pointer is purely stack-local and is not persisted into the state of the future/stream while the read/write is in progress. It's exclusively used only if LOCAL is returned and LOCAL is only returned immediately. If "BLOCKED" is returned then it's guaranteed that LOCAL won't be returned and the "local area" will be unused.

This continue to have no way to "punch a hole" in the stream where you can directly exchange language-level values, but I would consider that a future extension.

So, in summary:

• canon future.complete-local : [i32] -> [] - traps if there's no pending read/write locally, otherwise unblocks the pending operation
• canon stream.complete-local : [i32 i32] -> [] - traps if there's no pending read/write locally, otherwise unblocks the pending operation with the second value as the number of items that were read/written.
• canon future.{read,write} - grows a new argument which is a pointer to a "local area". The pointer must have alignment 4 and be valid for 4 bytes. Used to write the pointer given to the "other intrinsic"
• canon stream.{read,write} - same as future read/write with a new argument, but this argument is valid for 8 bytes as it has both the pointer and length written into it
• canon {future,stream}.{read,write} - these can all return LOCAL as a new return code where the "local area" is filled in and means that the paired operation is pending within the component. If desired the local area can be used to complete the operation. Upon completion canon {future,stream}.complete-local should be invoked.

All reads/writes are always through the canonical ABI. One end is "pure" canonical ABI at all times and both ends are prepared to do a "quick lift/lower" if LOCAL is returned with the returned value.

---

goal here is to get something reasonable working in the near-term time frame (aka not fully optimal and solving all problems for everyone) while not being a huge burden either. I think this should handle the case of intra-component reads/writes where differing ABI options trap but that's hopefully a "power user case for the future".

[lukewagner]

Great writeup! I think that is a workable solution. Let me propose a slight variation that has a lot of overlap, but tweaks a few things with, I think, less moving parts (perhaps far less, depending on what you think of the final sentence) while also allowing the full hole-punch.

So first, agreed on all these points:

• {stream,future}.new returns both readable and writable ends and lowering a stream/future always produces a fresh readable end.
• There's a new "local" synchronous case in {stream,futre}.{read,write} when "the other end" is local.
• There is a temporary trap in the "power user case for the future" case where two statically-linked wasm modules that aren't sharing the same ABI get the two ends of a stream.

So the variation is:

• By default, {stream,future}.{read,write} trap in the "local" case. Later we relax this so that it works in all cases (taking advantage of lazy lifting+lowering to avoid the otherwise weird interleaving questions).
• There's a new "local" option (immediate or argument) on {stream,future}.{read,write} that, in the "local" case, returns the index of the other side (with the high bits indicating the local case).

With this new local option, guest code would do the following:

• Call {stream,future}.{read,write} passing the "local" option. If this returns "local" and the other end's index:
  • Look up the index in the associated data structure (likely a dense array indexed by waitable index) that is I believe needed in any case.
  • If the index is "known" (has an initialized element in the array), cancel the pending read/write and switch to purely intra-wasm streaming.
  • If the index is not "known", perform a default (non-local) {stream,future}.{read,write} (which in the short term would immediately trap, but would later start working). If we're worried that this would allow latent bugs hidden behind the trap, the guest code could just trap itself.
• Alternatively, guest code that didn't want to even think about this could unconditionally call the default (non-local) {stream,future}.{read,write} and simply always trap in the same-component case.

What I like about this approach is that, in the final state, there are only two optimization levels: one that always copies, and one that enables full optimization (which, IIUC, requires learning the other side's index so you can take over the stream entirely).

Writing that final bullet made me think though: if we realllly wanted to be minimal for 0.3.0, we could just start with the final bullet (trap on same-component-instance-rendezvous) and then follow-up with the rest of it in 0.3.x. 🤷

[alexcrichton]

So far I've taken it as a constraint that for a workable solution we can't have the only option for same-instance-future/stream be trap, we've got to have some sort of hatch. Whether or not we should have that constraint I'm not sure, but assuming we stick to it personally I would prefer to not have either CABI or pure local streaming.

For me the problem with that is that it requires global tracking of handles both on the guest and the host. The guest has to keep track of all known handles (which breaks down in the polyglot case) and the host also has to do it to know which handle to give you. What I was thinking only requires the host to understand that each handle is in the same component instance. Additionally having to switch to purely local vs through the CABI forces the guest to deal with a much more complicated state machine.

For example let's say a read-end tries to read an active write happening elsewhere in the component. That returns "nope it's local, here's the index". Global maps all check out and the reader finds the write end. There's a few complications about what to do now:

• One option is the write-end's state machine is no longer purely driven by itself. The reader finishes the write for the writer and then wakes it up. That means pretty tight coordination and the writer has to account for the fact that its state could change remotely without it taking action.
• Another option is that the write end is just woken up saying "try again". This then deals with lots of complications where the read/write was supposed to complete but if you wake up the writer it might change its might and cancel the write and send the end to another component. This means that it's a complicated protocol to fully transition between the local/remote streaming cases.
• Currently with Rust bindings at least lifting is a destructive operation with respect to own<T>. That means, in the most general case, it's not possible to do purely in-component streaming because once the CABI write has started the original T is a "husk" that no longer works. Additionally a lift operation may perform allocations because Vec<T> in Rust is not the same memory layout as list<T> in the component model.

Personally I don't see a viable path forward at this time for the purely local component streaming case. I want to get it somewhat working to avoid having a brittle system where futures might trap if the writer happens to be in the same component, but beyond that I think it's going to be a lot of investment to get the purely local system working.

I understand though that the goal is to design for a better future with lazy lifting/lowering which may better handle the "I just want this to work" intra-component case. My worry is that the near-term assumption that "just return the other handle index" is going to not be too much work to get done is overly optimistic.

[lukewagner]

Whether or not we should have that constraint I'm not sure

The more I think about it, the more I think that it is likely that we can get away without having to support same-component streaming within the timeframe during which we also don't have other quite-important things (like error-context and splicing/forwarding). E.g., our motivating HTTP-proxy-composition scenarios shouldn't hit this and many of the "library of stream transformer" use cases (where you'd want to chain functions defined by a single imported instance) will naturally be host-implemented. And this can give us some time to property talk through these important state-machine questions you're raising. Maybe I'm wrong and if we land a trap for now it'll immediately blow up, but I'd like to have that data point before preemptively standardizing a mid-point optimization.

[alexcrichton]

Ok I think I'm willing to buy that, and so if we accept that the spec changes are:

• Creating a future/stream returns two handles.
• If a read/write operation on a future/stream "meets" in the same component, a trap happens.
• For now there's no way to test that a trap will happen.
• The plan is to see if this traps in practice, and design around the situation at hand.

The end result is simplified guest bindings with two types coming out of a new operation and no expectation to handle the "local" case where both ends are in the same component. Guest languages will document, however, that this trap condition is possible.

So only ABI changes to new, and only a semantic change with trapping on a intra-component read/write rendezvous.

[lukewagner]

Sounds right to me!