asm: introduce a new x64 assembler #10110

abrown · 2025-01-24T22:24:47Z

This is a first step to providing an external assembler for cranelift-codegen as described in #41. Each commit has further details, but the summary is that this direction would eventually move all assembler logic out into crates designed for easier checking (e.g., fuzzing).

abrown · 2025-01-25T00:42:02Z

cranelift/codegen/src/isa/x64/inst/emit.rs

+            // These values are transcribed from is happening in
+            // `SyntheticAmode::finalize`. This, plus the `Into` logic converting a
+            // `SyntheticAmode` to its external counterpart, are
+            let frame = state.frame_layout();
+            known_offsets[external::offsets::KEY_INCOMING_ARG] =
+                i32::try_from(frame.tail_args_size + frame.setup_area_size).unwrap();
+            known_offsets[external::offsets::KEY_SLOT_OFFSET] =
+                i32::try_from(frame.outgoing_args_size).unwrap();


@cfallin, I'm not a big fan of this KnownOffsetTable approach: it seems like we may want to just add appropriate CodeSink/MachBuffer methods to propagate this kind of thing.

Perhaps! Were you thinking we'd have specific trait methods for e.g. incoming_arg and slot_offset? Or that we'd still be generic over the set of known offsets but have a query method?

I'm totally fine with the latter, in fact it's probably even cleaner; for the former I'd be concerned about baking details of Cranelift's ABI impl into the lower-level library and would still want to follow the analogy of a real assembler letting one define symbolic constants and use them with specific values plugged in. Happy to look at proposed diff if you have one.

I agree, we should avoid Cranelift-izing this assembler as much as possible. I guess things I don't like about the known offset table approach is that (a) we rebuild this table for every emissions and (b) building it the way I'm doing it here feels quite fragile. How long before things get out of sync with the keys used during amode conversion?!

abrown · 2025-01-25T00:45:14Z

cranelift/filetests/filetests/isa/x64/simd-arith-avx.clif

@@ -1648,8 +1646,6 @@ block0(v0: i8x16, v1: i32):
 ;   addb %al, (%rax)
 ;   addb %al, (%rax)
 ;   addb %al, (%rax)
-;   addb %al, (%rax)
-;   addb %bh, %bh


Looking at this CLIF diffs, I see some potential problems: removed instructions, different immediates, potential add -> or miscompilations... This bears another look.

(Pretty sure those add -> or "miscompilations" were just bits changing in the constant pool... not sure why they're changing, though).

This change adds some initial logic implementing an external assembler for Cranelift's x64 backend, as proposed in RFC [bytecodealliance#41]. This adds two crates: - the `cranelift/assembler/meta` crate defines the instructions; to print out the defined instructions use `cargo run -p cranelift-assembler-meta` - the `cranelift/assembler` crate exposes the generated Rust code for those instructions; to see the path to the generated code use `cargo run -p cranelift-assembler` The assembler itself is straight-forward enough (modulo the code generation, of course); its integration into `cranelift-codegen` is what is most tricky about this change. Instructions that we will emit in the new assembler are contained in the `Inst::External` variant. This unfortunately increases the memory size of `Inst`, but only temporarily if we end up removing the extra `enum` indirection by adopting the new assembler wholesale. Another integration point is ISLE: we generate ISLE definitions and a Rust helper macro to make the external assembler instructions accessible to ISLE lowering. This change introduces some duplication: the encoding logic (e.g. for REX instructions) currently lives both in `cranelift-codegen` and the new assembler crate. The `Formatter` logic for the assembler `meta` crate is quite similar to the other `meta` crate. This minimal duplication felt worth the additional safety provided by the new assembler. The `cranelift-assembler` crate is fuzzable (see the `README.md`). It will generate instructions with randomized operands and compare their encoding and pretty-printed string to a known-good disassembler, currently `capstone`. This gives us confidence we previously didn't have regarding emission. In the future, we may want to think through how to fuzz (or otherwise check) the integration between `cranelift-codegen` and this new assembler level. [bytecodealliance#41]: bytecodealliance/rfcs#41

Using the new assembler's pretty-printing results in slightly different disassembly of compiled CLIF. This is because the assembler matches a certain configuration of `capstone`, causing the following obvious differences: - instructions with only two operands only print two operands; the original `MInst` instructions separate out the read-write operand into two separate operands (SSA-like) - the original instructions have some space padding after the instruction mnemonic, those from the new assembler do not This change uses the slightly new style as-is, but this is open for debate; we can change the configuration of `capstone` that we fuzz against. My only preferences would be to (1) retain some way to visually distinguish the new assembler instructions in the disassembly (temporarily, for debugging) and (2) eventually transition to pretty-printing instructions in Intel-style (`rw, r`) instead of the current (`r, rw`).

Though it is likely that `rustfmt` is present in a Rust environment, some CI tasks do not have this tool installed. To handle this case (plus the chance that other Wasmtime builds are similar), this change skips formatting with a `stderr` warning when `rustfmt` fails.

In order to satisfy `ci/publish.rs`, it would appear that we need to use a version that matches the rest of the Cranelift crates.

cfallin

This looks really really good -- thanks for spawning this idea and working to build it out!

Most of my feedback you already got offline as this was being built and the general shape is therefore what I'd expect and am overall quite happy with. A bunch of nits below but nothing major -- let's get this merged and we can iterate and start the migration process.

Thanks again!

cranelift/assembler-x64/Cargo.toml

cranelift/assembler-x64/meta/src/dsl.rs

cranelift/assembler-x64/meta/src/dsl/encoding.rs

cfallin · 2025-01-29T18:22:38Z

cranelift/assembler-x64/src/imm.rs

+
+/// This helper function prints the hexadecimal representation of the immediate
+/// value, but only if the value is greater than or equal to 10. This is
+/// necessary to match how Capstone pretty-prints immediate values.


Egads -- is there a way to configure Capstone to always print in decimal or hex? (Necessary evil otherwise, not your fault, just ugly)

cfallin · 2025-01-29T18:25:28Z

cranelift/assembler-x64/src/reg.rs

+/// This is due to avoid special cases of REX encodings, see Intel SDM Vol. 2A,
+/// table 2-5.
+#[derive(Clone, Copy, Debug)]
+pub struct MinusRsp<R: AsReg>(R);


MinusRsp evokes some sort of subtraction operation in my mind; maybe NonRspGpr or something like that?

cfallin · 2025-01-29T18:29:27Z

cranelift/codegen/src/isa/x64/inst/emit.rs

+            let mut known_offsets = [0, 0];
+            // These values are transcribed from is happening in
+            // `SyntheticAmode::finalize`. This, plus the `Into` logic converting a
+            // `SyntheticAmode` to its external counterpart, are


"from is" and ending with "are..." -- incomplete sentence?

cfallin · 2025-01-29T18:32:04Z

cranelift/codegen/src/isa/x64/inst/emit.rs

+            // These values are transcribed from is happening in
+            // `SyntheticAmode::finalize`. This, plus the `Into` logic converting a
+            // `SyntheticAmode` to its external counterpart, are
+            let frame = state.frame_layout();
+            known_offsets[external::offsets::KEY_INCOMING_ARG] =
+                i32::try_from(frame.tail_args_size + frame.setup_area_size).unwrap();
+            known_offsets[external::offsets::KEY_SLOT_OFFSET] =
+                i32::try_from(frame.outgoing_args_size).unwrap();


Perhaps! Were you thinking we'd have specific trait methods for e.g. incoming_arg and slot_offset? Or that we'd still be generic over the set of known offsets but have a query method?

I'm totally fine with the latter, in fact it's probably even cleaner; for the former I'd be concerned about baking details of Cranelift's ABI impl into the lower-level library and would still want to follow the analogy of a real assembler letting one define symbolic constants and use them with specific values plugged in. Happy to look at proposed diff if you have one.

cfallin · 2025-01-29T18:34:52Z

cranelift/codegen/src/isa/x64/pcc.rs

@@ -980,6 +980,8 @@ pub(crate) fn check(
        Inst::Unwind { .. } | Inst::DummyUse { .. } => Ok(()),

        Inst::StackSwitchBasic { .. } => Err(PccError::UnimplementedInst),
+
+        Inst::External { .. } => Ok(()), // TODO: unsure what to do about this!


Totally fine to do Err(PccError::UnimplementedInst) for now -- I'll need to go and update this once we've moved instructions over that PCC cares about, but PCC isn't fuzzed at the moment anyway until I can make an update/fix pass...

abrown force-pushed the assembler-upstream branch 2 times, most recently from d209114 to 9140ca2 Compare January 24, 2025 22:59

github-actions bot added the cranelift Issues related to the Cranelift code generator label Jan 24, 2025

abrown force-pushed the assembler-upstream branch 2 times, most recently from 943fb8e to d694e80 Compare January 25, 2025 00:39

abrown commented Jan 25, 2025

View reviewed changes

abrown force-pushed the assembler-upstream branch from d694e80 to 574176f Compare January 25, 2025 00:46

abrown added 5 commits January 24, 2025 16:47

vet: audit arbtest for use as a dev-dependency

0268ef9

ci: make assembler crates publishable

f6e4f1d

In order to satisfy `ci/publish.rs`, it would appear that we need to use a version that matches the rest of the Cranelift crates.

abrown force-pushed the assembler-upstream branch from 574176f to f6e4f1d Compare January 25, 2025 00:47

abrown requested a review from cfallin January 25, 2025 01:06

abrown marked this pull request as ready for review January 25, 2025 01:06

abrown requested review from a team as code owners January 25, 2025 01:06

cfallin approved these changes Jan 29, 2025

View reviewed changes

abrown added 8 commits January 30, 2025 16:24

review: use Cargo workspace values

594a3ca

review: document Inst, move Inst::name

c3e28c4

review: clarify 'earlier' doc comment

ff778af

review: document multi-byte opcodes

043f501

review: document Rex builder methods

1fd0e1d

review: document encoding rules

04aeb44

review: clarify 'bits' -> 'width'

6084920

review: clarify confusing legacy prefixes

fab310a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

asm: introduce a new x64 assembler #10110

asm: introduce a new x64 assembler #10110

abrown commented Jan 24, 2025

abrown Jan 25, 2025

cfallin Jan 29, 2025

abrown Jan 31, 2025

abrown Jan 25, 2025 •

edited

Loading

abrown Jan 25, 2025

cfallin left a comment

cfallin Jan 29, 2025

cfallin Jan 29, 2025

cfallin Jan 29, 2025

cfallin Jan 29, 2025

cfallin Jan 29, 2025

asm: introduce a new x64 assembler #10110

Are you sure you want to change the base?

asm: introduce a new x64 assembler #10110

Conversation

abrown commented Jan 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abrown Jan 25, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfallin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abrown Jan 25, 2025 •

edited

Loading