Relaxed SIMD

[ngzhian]

Relaxed SIMD adds a set of useful instructions that introduce local non-determinism where the results of the instructions may vary based on hardware support.

The SIMD proposal focuses on getting a set of SIMD instructions that will speed up real world use cases while staying true to the deterministic core of the language. However, there are instructions that can unlock even more performance, but due to the architecture-dependent semantics, were not included. These instructions include:

• Fused Multiply Add (single rounding if hardware supports it, double rounding if not)
• Approximate reciprocal/reciprocal sqrt
• Relaxed Swizzle (implementation defined out of bounds behavior)
• Relaxed Rounding Q-format Multiplication (optional saturation)

These instructions have been suggested in multiple places: FMA (1, 2, 3), approximate reciprocal/reciprocal sqrt (1). Such instructions have also been mentioned as part of future features.

There is a soft dependency on feature-detection proposal, which will allow code to determine if certain instructions are supported by the hardware and these instructions can be safely relied on.

Non-determinism: The non-determinism in this proposal is limited to the result of an individual instruction and and is consistent across runs. There are no global control or flags involved. This means that given the following pseudocode:

w = fma(x, y, z)

w can have different values depending on available hardware support. Multiple usages of the instruction will return the same result w, so the instruction is internally consistent.

Initial prototypes indicate performance improvement of ~30% on modern CPU architectures. The alternative, which is to provide a deterministic FMA result using emulation, will be too slow to be of any use.

Potential extension: introduce a relaxed mode for existing SIMD instructions. Such a mode would be tied to the feature-detection proposal, where if relaxed-mode is supported, the existing SIMD instructions will be have non-deterministic behavior, e.g. NaN canonicalization, FP IEEE compliance modes used by developers (e.g. no-honor-inf, no-signed-zeros, no-trapping-math..) .

Keywords (for SEO): Fast SIMD

Co-champions: Marat Dukhan (@Maratyszcza) and Zhi An Ng (@ngzhian)

[arunetm]

Thanks @ngzhian @Maratyszcza. This is super-useful.

It will be helpful if instructions as part of this extension are not limited to this top 5. We had evaluated Float min/max, float-int conversions, etc to be able to offer significant performance gains depending on platforms. Having these as new instruction variants than a relaxed-simd-mode extension (mentioned above) will help to minimize potential non-determinism introduced by engines.
Beyond nan-propagation, there are a few other scenarios that may benefit from relaxed semantics, especially when the FP IEEE compliance modes are used by developers (e.g. no-honor-inf, no-signed-zeros, no-trapping-math..) It will be useful if we can consider those under this umbrella as well.

[ngzhian]

It will be helpful if instructions as part of this extension are not limited to this top 5.

Yes, that list is not exhaustive, I hope we can come up with more when we eventually have a repo (similar to how we proposed and merged instructions for SIMD).

Beyond nan-propagation, there are a few other scenarios that may benefit from relaxed semantics, especially when the FP IEEE compliance modes are used by developers (e.g. no-honor-inf, no-signed-zeros, no-trapping-math..) It will be useful if we can consider those under this umbrella as well.

Good idea, I'll add this snippet to the description, thanks.

[arunetm]

Thanks. Related: #1393 (comment)

I hope we can come up with more when we eventually have a repo

Is there a near-term plan to have the CG phase 0/1 poll?

[ngzhian]

Is there a near-term plan to have the CG phase 0/1 poll?
Yes, once we get more comments, perhaps we can poll at the March 16 CG meeting.

[sunfishcode]

A tricky thing about Relaxed SIMD and related operators is the meaning of "consistent across runs". It's obviously valuable for a "program" to be able to assume that rounding is consistent across a "run", to avoid discontinuities etc., but in wasm in general, it's not always clear what the scope of a "run" is, for example for long-lived suspended and resumed instances, or instance graphs distributed across multiple underlying hosts. I have an idea for how to avoid this, and I'm curious what others think:

Introduce a new opaque type, fpenv, representing information about the host floating-point environment, which would be passed to operators like qfma as an operand. This could evolve in various ways in the future, but for now, it would just include flags like "is fma fast?".

Initially, the only way to obtain an fpenv value would be to import a global variable with type fpenv:

   (global $fp (import "host.fp" "default") fpenv)
   
   (func $foo (param f32) (param f32) (param f32) (result f32)
     local.get 0
     local.get 1
     local.get 2
     global.get $fp
     f32.qfma
   )

(The names "host.fp" and "default" here are just for illustration; this is something we'd need to figure out.)

Instead of saying qfma itself is nondeterministic, we'd say the value of the fpenv you import is nondeterministic. That would make the "same across a run" property explicit in the code. If one wants to guarantee that two instances have the same value (eg. a main program instance and a library instance), one module could import it and then re-export it, and the other could import it from the first (toolchains could arrange this automatically).

In typical implementations, fpenv wouldn't be a dynamic value. Many hosts would have only one possible fpenv value, so all instructions that interact with fpenv values could be no-ops.

A downside is wasm code size; the import and global.get instructions take some space. However, the impact should be relatively small, since instructions that need an fpenv value are likely to be relatively infrequent in whole modules.

[tlively]

It's obviously valuable for a "program" to be able to assume that rounding is consistent across a "run", to avoid discontinuities etc.

I agree with this, but I'm not convinced it's worth guaranteeing this kind of strong internal consistency in the spec. We could leave it up to distributed and migratory engines to choose for themselves whether to provide this guarantee, and engines that are not distributed or migratory would be able to provide it without doing anything extra. The purpose of these instructions is to dramatically improve performance, but engines with strong determinism requirements won't be able to realize any performance improvement from them. In practice, I expect engines with such determinism requirements to not want to implement these instructions to begin with, so introducing extra complexity to make them deterministic would not be worth it.

[sunfishcode]

The purpose of these instructions is to dramatically improve performance, but engines with strong determinism requirements won't be able to realize any performance improvement from them. In practice, I expect engines with such determinism requirements to not want to implement these instructions to begin with, so introducing extra complexity to make them deterministic would not be worth it.

FMA seems to be a counterexample to this; it's available on lots of CPUs these days (and that link is out of date; there are many more now), so for example, many server environments today can comfortably depend on having FMA. It has wide applicability, including in domains where both determinism and distributed compute are important (linear algebra over tiled datasets). And it delivers major speedups (the ~30% number mentioned above).

[ngzhian]

It's obviously valuable for a "program" to be able to assume that rounding is consistent across...

The non-determinism in this proposal is limited to the result of an individual instruction and and is consistent across runs...

Are there cases where, if the instruction is not "consistent across runs" (for some definition), it will still be useful, and not confusing?

I'm thinking of defining it like:

fma(x, y, z) = oneof { round(x + y * z), round(round(x+y) * z) }

The determinism is local, and for this particular example there are always only 2 possible cases. But it isn't consistent, in that if you have the distributed execution, one host can choose to do a single rounding and another host can choose to do two roundings.

If the application is robust to such differences, then they get the most performance out of this.

[sunfishcode]

Are there cases where, if the instruction is not "consistent across runs" (for some definition), it will still be useful, and not confusing?

There surely do exist applications that are robust to FMA rounding differently each time it's invoked. But there are also applications where absolute precision isn't important, but avoiding discontinuities is. For example, in many graphics programs, it may not be noticeable if the position or color of a particular object is slightly different from what a fully precise computation might show, but it may be noticeable if there's a bump in a line or a boundary in a gradient.

I've seen GPU drivers split FMAs into discrete multiplies and adds in places where they can fit just the multiply or the add into the hardware pipeline in a particular place in the code, and I've seen actual applications be broken as a result. If we want consistency, we should say so in the spec.

[tlively]

What if we solve the problem in a different direction by instead clarifying what we mean by "run" in the spec? We could add language to the effect of "FMA may have this or that behavior, but an instance may not observe both behaviors and all instances in a store must observe the same behavior." Again, for non-distributed, non-migratory engines, this imposes no additional burden. Distributed/migratory engines would still have to do whatever they would do with the explicit fpenv, but from the producer and spec perspective, this would be simpler.

[sunfishcode]

What if we solve the problem in a different direction by instead clarifying what we mean by "run" in the spec? We could add language to the effect of "FMA may have this or that behavior, but an instance may not observe both behaviors and all instances in a store must observe the same behavior." Again, for non-distributed, non-migratory engines, this imposes no additional burden. Distributed/migratory engines would still have to do whatever they would do with the explicit fpenv, but from the producer and spec perspective, this would be simpler.

The store is abstract, just a way for the spec to talk about entities that can be linked. Can a store span suspend/resume cycles? Can it span networks? The spec just says if you can link exports to imports, it's a store, which gives embedders a lot of flexibility. If we start using the store to hold miscellaneous configuration information, it would give the store an identity, making these questions more complex.

The point of fpenv is that engines can't infer it on their own. Producers know their intent, such as "this library needs the same floating-point configuration as the program it's linked to", and fpenv lets the state their intent.

[tlively]

I see your point about the store being an imperfect abstraction for this use case. If an engine snapshots an instance's memory and reinstantiates the module with that snapshotted memory on a different machine, it seems that you could call that a different store but still break the program if FMA semantics change.

I guess I'm not convinced that letting producers express fine-grained intent here is useful enough to be worth this extra complexity, though. I can see that a use case could be constructed, but I also know that the users I work with who really want to use FMA won't need this. I'm of course biased, though, because I only work closely with Web users, for whom this wouldn't make a difference. Do you know of users who are eager for this fine-grained control? If there are none now, perhaps we could either make no guarantees or find a way to specify coarse-grained guarantees about semantic consistency in this proposal. Assuming there are no users now, I would be happy to revisit this and do another version with finer-grained control in the future when there are users ready to take advantage of that.

[sunfishcode]

One of the interesting properties of wasm is its virtualizability. All interactions with the outside world go through imports and exports, so it's straightforward to have WebAssembly instances with no concept of "the machine I'm on" as a distinct identity they can interact with. With WASI, we're working on an ecosystem where programs don't know about "the filesystem namespace of the machine I'm on" or "the local network configuration of the machine I'm on" (substitute in "the container I'm in" or "the VM I'm in" as needed ;-)).

This means "the machine I'm on" could change over time, and "the machine I'm on" could be different from "the machine other instances I'm linked to are on". We can do this, without prearranged configuration, because the relationships between instances are completely described in their imports and exports. I myself and others are building on systems which will take advantage of this property in general.

There are cases where we can get by without this property. For example with FMA, if we're ok limiting our scope to just servers, then we can maybe just depend on FMA being available everywhere. However, on one hand, I can't guarantee we'll always limit our scope to just servers. And on the other, the proposal here already has more than just FMA, and more things may be added in the future.

I understand this is adding complexity up front, but my concern is that if we don't preserve this property of wasm, it will be difficult to re-introduce. And it doesn't seem like it's that complex for producers to produce or for minimal consumers to ignore.

[RossTate]

How is this addressed for NaNs in the f32/f64 operations?

[sunfishcode]

The bits of a NaN produced by an f32/f64 operation are simply nondeterministic, within a few constraints, so they don't imply any state. And in practice, it's rare for programs to care about the bits of NaNs in ways that aren't covered by the constraints.