threads: add shared heap types (#1600)

* threads: add `shared` heap types

The shared-everything-thread [proposal] specifies that all abstract heap
types can be marked `shared` by prepending them with the 0x65 prefix
byte. While we wait for clarification on what this precisely means, I
chose to implement the "extra byte" version, which will result in larger
binaries that use `shared` types.

The key change here is that `wasmparser::HeapType` now groups all of the
abstract heap types (e.g., `any`) into a
`wasmparser::HeapType::Abstract` variant, which allows us to mark those
as `shared` in a single place. This results in a cascade of changes
elsewhere, most of which are pretty harmless (an extra, internal `match`
on the `ty` of the `Abstract` variant). The real business left
unfinished by this commit is _what_ to do with the `shared` field in all
of these locations; for now I've just noted those as TODOs to get a
review on this approach so far before forging ahead.

[proposal]: https://github.com/WebAssembly/shared-everything-threads
[#64]: WebAssembly/shared-everything-threads#64

* threads: propagate `shared` field to `wasm_encoder`

This continues the work of the previous commit by propagating the
`shared` field of abstract heap types through to `wasm_encoder`, with
uses in various other crates. As before, the boundary at which we cannot
yet handle sharedness is marked with TODO.

* threads: propagate `shared` into `wast`

This finally extends `shared` into the heap types defined in `wast`. As
before, a distinction is drawn between abstract and concrete heap
types.

* review: refactor reading of short-hand reftypes

* review: run CI-specific `rustfmt` invocation

* review: use short-hand `RefType` constant for brevity

* review: use `Into` for wasmparser-to-wasm-encoder conversion

* review: document which proposal `shared` appears in

* review: fix typo, 'eqf' -> 'eq'

* review: remove unnecessary TODO

* review: link to shared-ness issue for reference conversions

* review: fix typo

* review: add `wasmparser` feature for `Into` conversions

* review: use type shorthand in wasm-smith

* review: use more `Into` conversions

* review: add aliases for heap types

* review: add clarity parentheses

* review: add and use more heap type aliases (in wasmparser)

* review: use aliases... more

* review: integrate with upstream

* review: re-add full explicit matching in subtype check

* fix: re-run rustfmt

* review: handle shared peeking in wast

* threads: check reference types for `shared`-ness

In previous commits, we were unable to determine if a reference type was
shared because concrete heap types point to a type elsewhere in the
module that we weren't able to retrieve at the time the check was made.
This change fixes that by plumbing a `TypeList` through to the right
place and implementing `TypeList::is_shared`. This will still fail with
a `todo!()` at the boundary of where shared-ness is implemented; once
composite types gain shared flags that `todo!()` should go away.

* threads: fix parsing ambiguities

When parsing a shared global in the WebAssembly text format, one might
run across the following text: `(global (shared anyref))`. In the
current paradigm, this should parse to a `shared` global that contains
an invalid `anyref` (i.e., `(ref null any)` in long-form, which is
unshared); this should be parseable but result in a validation error.
The correct form is `(global (shared (shared anyref)))`.

This change fixes several issues related to this as well as refactoring
some of the "short-hand"/"long-hand" parsing code.

* threads: add heap type tests

This change adds generated tests for all of the abstract heap types when
they are `shared` and placed in `shared` globals. Since globals are the
only thing we can mark shared now that can touch these types, they are a
good vehicle for testing the parsing, encoding, and validation of all of
this. Eventually the same must be done for concrete heap types once
composite types (`func`, `struct`, `array`) can be marked shared.

Author: abrown

crates/wasm-encoder/src/core/types.rs

@@ -344,33 +344,91 @@ pub struct RefType {
 }
 
 impl RefType {
-    /// Alias for the `funcref` type in WebAssembly
+    /// Alias for the `anyref` type in WebAssembly.
+    pub const ANYREF: RefType = RefType {
+        nullable: true,
+        heap_type: HeapType::Abstract {
+            shared: false,
+            ty: AbstractHeapType::Any,
+        },
+    };
+
+    /// Alias for the `anyref` type in WebAssembly.
+    pub const EQREF: RefType = RefType {
+        nullable: true,
+        heap_type: HeapType::Abstract {
+            shared: false,
+            ty: AbstractHeapType::Eq,
+        },
+    };
+
+    /// Alias for the `funcref` type in WebAssembly.
     pub const FUNCREF: RefType = RefType {
         nullable: true,
-        heap_type: HeapType::Func,
+        heap_type: HeapType::Abstract {
+            shared: false,
+            ty: AbstractHeapType::Func,
+        },
     };
 
-    /// Alias for the `externref` type in WebAssembly
+    /// Alias for the `externref` type in WebAssembly.
     pub const EXTERNREF: RefType = RefType {
         nullable: true,
-        heap_type: HeapType::Extern,
+        heap_type: HeapType::Abstract {
+            shared: false,
+            ty: AbstractHeapType::Extern,
+        },
     };
 
-    /// Alias for the `exnref` type in WebAssembly
+    /// Alias for the `i31ref` type in WebAssembly.
+    pub const I31REF: RefType = RefType {
+        nullable: true,
+        heap_type: HeapType::Abstract {
+            shared: false,
+            ty: AbstractHeapType::I31,
+        },
+    };
+
+    /// Alias for the `arrayref` type in WebAssembly.
+    pub const ARRAYREF: RefType = RefType {
+        nullable: true,
+        heap_type: HeapType::Abstract {
+            shared: false,
+            ty: AbstractHeapType::Array,
+        },
+    };
+
+    /// Alias for the `exnref` type in WebAssembly.
     pub const EXNREF: RefType = RefType {
         nullable: true,
-        heap_type: HeapType::Exn,
+        heap_type: HeapType::Abstract {
+            shared: false,
+            ty: AbstractHeapType::Exn,
+        },
     };
+
+    /// Set the nullability of this reference type.
+    pub fn nullable(mut self, nullable: bool) -> Self {
+        self.nullable = nullable;
+        self
+    }
 }
 
 impl Encode for RefType {
     fn encode(&self, sink: &mut Vec<u8>) {
         if self.nullable {
             // Favor the original encodings of `funcref` and `externref` where
-            // possible
+            // possible.
+            use AbstractHeapType::*;
             match self.heap_type {
-                HeapType::Func => return sink.push(0x70),
-                HeapType::Extern => return sink.push(0x6f),
+                HeapType::Abstract {
+                    shared: false,
+                    ty: Func,
+                } => return sink.push(0x70),
+                HeapType::Abstract {
+                    shared: false,
+                    ty: Extern,
+                } => return sink.push(0x6f),
                 _ => {}
             }
         }
@@ -405,6 +463,78 @@ impl From<RefType> for ValType {
 /// Part of the function references proposal.
 #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Ord, PartialOrd)]
 pub enum HeapType {
+    /// An abstract heap type; e.g., `anyref`.
+    Abstract {
+        /// Whether the type is shared.
+        shared: bool,
+        /// The actual heap type.
+        ty: AbstractHeapType,
+    },
+
+    /// A concrete Wasm-defined type at the given index.
+    Concrete(u32),
+}
+
+impl HeapType {
+    /// Alias for the unshared `any` heap type.
+    pub const ANY: Self = Self::Abstract {
+        shared: false,
+        ty: AbstractHeapType::Any,
+    };
+
+    /// Alias for the unshared `func` heap type.
+    pub const FUNC: Self = Self::Abstract {
+        shared: false,
+        ty: AbstractHeapType::Func,
+    };
+
+    /// Alias for the unshared `extern` heap type.
+    pub const EXTERN: Self = Self::Abstract {
+        shared: false,
+        ty: AbstractHeapType::Extern,
+    };
+
+    /// Alias for the unshared `i31` heap type.
+    pub const I31: Self = Self::Abstract {
+        shared: false,
+        ty: AbstractHeapType::I31,
+    };
+}
+
+impl Encode for HeapType {
+    fn encode(&self, sink: &mut Vec<u8>) {
+        match self {
+            HeapType::Abstract { shared, ty } => {
+                if *shared {
+                    sink.push(0x65);
+                }
+                ty.encode(sink);
+            }
+            // Note that this is encoded as a signed type rather than unsigned
+            // as it's decoded as an s33
+            HeapType::Concrete(i) => i64::from(*i).encode(sink),
+        }
+    }
+}
+
+#[cfg(feature = "wasmparser")]
+impl TryFrom<wasmparser::HeapType> for HeapType {
+    type Error = ();
+
+    fn try_from(heap_type: wasmparser::HeapType) -> Result<Self, Self::Error> {
+        Ok(match heap_type {
+            wasmparser::HeapType::Concrete(i) => HeapType::Concrete(i.as_module_index().ok_or(())?),
+            wasmparser::HeapType::Abstract { shared, ty } => HeapType::Abstract {
+                shared,
+                ty: ty.into(),
+            },
+        })
+    }
+}
+
+/// An abstract heap type.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Ord, PartialOrd)]
+pub enum AbstractHeapType {
     /// Untyped (any) function.
     Func,
 
@@ -455,53 +585,46 @@ pub enum HeapType {
 
     /// The abstract `noexn` heap type.
     NoExn,
-
-    /// A concrete Wasm-defined type at the given index.
-    Concrete(u32),
 }
 
-impl Encode for HeapType {
+impl Encode for AbstractHeapType {
     fn encode(&self, sink: &mut Vec<u8>) {
+        use AbstractHeapType::*;
         match self {
-            HeapType::Func => sink.push(0x70),
-            HeapType::Extern => sink.push(0x6F),
-            HeapType::Any => sink.push(0x6E),
-            HeapType::None => sink.push(0x71),
-            HeapType::NoExtern => sink.push(0x72),
-            HeapType::NoFunc => sink.push(0x73),
-            HeapType::Eq => sink.push(0x6D),
-            HeapType::Struct => sink.push(0x6B),
-            HeapType::Array => sink.push(0x6A),
-            HeapType::I31 => sink.push(0x6C),
-            HeapType::Exn => sink.push(0x69),
-            HeapType::NoExn => sink.push(0x74),
-            // Note that this is encoded as a signed type rather than unsigned
-            // as it's decoded as an s33
-            HeapType::Concrete(i) => i64::from(*i).encode(sink),
+            Func => sink.push(0x70),
+            Extern => sink.push(0x6F),
+            Any => sink.push(0x6E),
+            None => sink.push(0x71),
+            NoExtern => sink.push(0x72),
+            NoFunc => sink.push(0x73),
+            Eq => sink.push(0x6D),
+            Struct => sink.push(0x6B),
+            Array => sink.push(0x6A),
+            I31 => sink.push(0x6C),
+            Exn => sink.push(0x69),
+            NoExn => sink.push(0x74),
         }
     }
 }
 
 #[cfg(feature = "wasmparser")]
-impl TryFrom<wasmparser::HeapType> for HeapType {
-    type Error = ();
-
-    fn try_from(heap_type: wasmparser::HeapType) -> Result<Self, Self::Error> {
-        Ok(match heap_type {
-            wasmparser::HeapType::Concrete(i) => HeapType::Concrete(i.as_module_index().ok_or(())?),
-            wasmparser::HeapType::Func => HeapType::Func,
-            wasmparser::HeapType::Extern => HeapType::Extern,
-            wasmparser::HeapType::Any => HeapType::Any,
-            wasmparser::HeapType::None => HeapType::None,
-            wasmparser::HeapType::NoExtern => HeapType::NoExtern,
-            wasmparser::HeapType::NoFunc => HeapType::NoFunc,
-            wasmparser::HeapType::Eq => HeapType::Eq,
-            wasmparser::HeapType::Struct => HeapType::Struct,
-            wasmparser::HeapType::Array => HeapType::Array,
-            wasmparser::HeapType::I31 => HeapType::I31,
-            wasmparser::HeapType::Exn => HeapType::Exn,
-            wasmparser::HeapType::NoExn => HeapType::NoExn,
-        })
+impl From<wasmparser::AbstractHeapType> for AbstractHeapType {
+    fn from(value: wasmparser::AbstractHeapType) -> Self {
+        use wasmparser::AbstractHeapType::*;
+        match value {
+            Func => AbstractHeapType::Func,
+            Extern => AbstractHeapType::Extern,
+            Any => AbstractHeapType::Any,
+            None => AbstractHeapType::None,
+            NoExtern => AbstractHeapType::NoExtern,
+            NoFunc => AbstractHeapType::NoFunc,
+            Eq => AbstractHeapType::Eq,
+            Struct => AbstractHeapType::Struct,
+            Array => AbstractHeapType::Array,
+            I31 => AbstractHeapType::I31,
+            Exn => AbstractHeapType::Exn,
+            NoExn => AbstractHeapType::NoExn,
+        }
     }
 }
 

crates/wasm-mutate/Cargo.toml

@@ -14,7 +14,7 @@ workspace = true
 clap = { workspace = true, optional = true }
 thiserror = "1.0.28"
 wasmparser = { workspace = true }
-wasm-encoder = { workspace = true }
+wasm-encoder = { workspace = true, features = ["wasmparser"] }
 rand = { workspace = true }
 log = { workspace = true }
 egg = "0.6.0"

crates/wasm-mutate/src/module.rs

@@ -80,19 +80,11 @@ pub fn map_ref_type(ref_ty: wasmparser::RefType) -> Result<RefType> {
     Ok(RefType {
         nullable: ref_ty.is_nullable(),
         heap_type: match ref_ty.heap_type() {
-            wasmparser::HeapType::Func => HeapType::Func,
-            wasmparser::HeapType::Extern => HeapType::Extern,
-            wasmparser::HeapType::Any => HeapType::Any,
-            wasmparser::HeapType::None => HeapType::None,
-            wasmparser::HeapType::NoExtern => HeapType::NoExtern,
-            wasmparser::HeapType::NoFunc => HeapType::NoFunc,
-            wasmparser::HeapType::Eq => HeapType::Eq,
-            wasmparser::HeapType::Struct => HeapType::Struct,
-            wasmparser::HeapType::Array => HeapType::Array,
-            wasmparser::HeapType::I31 => HeapType::I31,
-            wasmparser::HeapType::Exn => HeapType::Exn,
-            wasmparser::HeapType::NoExn => HeapType::NoExn,
             wasmparser::HeapType::Concrete(i) => HeapType::Concrete(i.as_module_index().unwrap()),
+            wasmparser::HeapType::Abstract { shared, ty } => {
+                let ty = ty.into();
+                HeapType::Abstract { shared, ty }
+            }
         },
     })
 }

crates/wasm-mutate/src/mutators/add_function.rs

@@ -4,7 +4,7 @@ use super::Mutator;
 use crate::module::{PrimitiveTypeInfo, TypeInfo};
 use crate::{Result, WasmMutate};
 use rand::Rng;
-use wasm_encoder::{HeapType, Instruction, Module};
+use wasm_encoder::{AbstractHeapType, HeapType, Instruction, Module};
 
 /// Mutator that adds new, empty functions to a Wasm module.
 #[derive(Clone, Copy)]
@@ -62,10 +62,16 @@ impl Mutator for AddFunctionMutator {
                     func.instruction(&Instruction::V128Const(0));
                 }
                 PrimitiveTypeInfo::FuncRef => {
-                    func.instruction(&Instruction::RefNull(HeapType::Func));
+                    func.instruction(&Instruction::RefNull(HeapType::Abstract {
+                        shared: false,
+                        ty: AbstractHeapType::Func,
+                    }));
                 }
                 PrimitiveTypeInfo::ExternRef => {
-                    func.instruction(&Instruction::RefNull(HeapType::Extern));
+                    func.instruction(&Instruction::RefNull(HeapType::Abstract {
+                        shared: false,
+                        ty: AbstractHeapType::Extern,
+                    }));
                 }
                 PrimitiveTypeInfo::Empty => unreachable!(),
             }

crates/wasm-mutate/src/mutators/modify_const_exprs.rs

@@ -124,8 +124,14 @@ impl<'cfg, 'wasm> Translator for InitTranslator<'cfg, 'wasm> {
                 } else {
                     f64::from_bits(self.config.rng().gen())
                 }),
-                T::FUNCREF => CE::ref_null(wasm_encoder::HeapType::Func),
-                T::EXTERNREF => CE::ref_null(wasm_encoder::HeapType::Extern),
+                T::FUNCREF => CE::ref_null(wasm_encoder::HeapType::Abstract {
+                    shared: false,
+                    ty: wasm_encoder::AbstractHeapType::Func,
+                }),
+                T::EXTERNREF => CE::ref_null(wasm_encoder::HeapType::Abstract {
+                    shared: false,
+                    ty: wasm_encoder::AbstractHeapType::Func,
+                }),
                 T::Ref(_) => unimplemented!(),
             }
         } else {

crates/wasm-mutate/src/mutators/peephole/dfg.rs

@@ -758,12 +758,12 @@ impl<'a> DFGBuilder {
                 }
 
                 Operator::RefNull {
-                    hty: wasmparser::HeapType::Extern,
+                    hty: wasmparser::HeapType::EXTERN,
                 } => {
                     self.push_node(Lang::RefNull(RefType::Extern), idx);
                 }
                 Operator::RefNull {
-                    hty: wasmparser::HeapType::Func,
+                    hty: wasmparser::HeapType::FUNC,
                 } => {
                     self.push_node(Lang::RefNull(RefType::Func), idx);
                 }

crates/wasm-mutate/src/mutators/peephole/eggsy/lang.rs

@@ -3,7 +3,7 @@
 use egg::Id;
 use std::fmt::{self, Display};
 use std::str::FromStr;
-use wasm_encoder::HeapType;
+use wasm_encoder::{AbstractHeapType, HeapType};
 
 /// This is a macro used to define the `Lang` enum.
 ///
@@ -1072,8 +1072,14 @@ pub enum RefType {
 impl From<RefType> for HeapType {
     fn from(rt: RefType) -> Self {
         match rt {
-            RefType::Func => HeapType::Func,
-            RefType::Extern => HeapType::Extern,
+            RefType::Func => HeapType::Abstract {
+                shared: false,
+                ty: AbstractHeapType::Func,
+            },
+            RefType::Extern => HeapType::Abstract {
+                shared: false,
+                ty: AbstractHeapType::Extern,
+            },
         }
     }
 }

crates/wasm-mutate/src/mutators/snip_function.rs

@@ -61,10 +61,10 @@ impl Mutator for SnipMutator {
                                 f.instruction(&Instruction::V128Const(0));
                             }
                             PrimitiveTypeInfo::FuncRef => {
-                                f.instruction(&Instruction::RefNull(HeapType::Func));
+                                f.instruction(&Instruction::RefNull(HeapType::FUNC));
                             }
                             PrimitiveTypeInfo::ExternRef => {
-                                f.instruction(&Instruction::RefNull(HeapType::Extern));
+                                f.instruction(&Instruction::RefNull(HeapType::EXTERN));
                             }
                             PrimitiveTypeInfo::Empty => {
                                 unreachable!()