feat: Testing via RISCOF #175
Conversation
|
Would someone please comment on this PR? We want to make sure essential tests keep running, and this adds a standard testing interface that will allow us to do this. I'd be happy to iterate with your guidance. Thanks! |
|
@shamatar do you have a comment? |
|
@codygunton The problem is that prover in "main" is already previous-version, and the version used for ethproofs (in "dev" branch) uses different risc-v simulator implementation (that lives here https://github.com/matter-labs/zksync-airbender/tree/dev/riscv_transpiler) that preprocessed the bytecode. New simulator for now lacks tracing/flamegraph generation features, and some other conveniences like "cycle markers". I do not know which of them do you need for this testing framework to work to be honest, and I can not predict difficulty of the extension. From our side we will only work on such features (flamegraphs/etc) in the new version only when our internal proving will migrate to it (zksync-os related changes). Even though the "dev" branch still has this old risc-v simulator code as a legacy, it's not longer relevant for real proving |
|
Understood, I will update my work to target the dev branch |
Add support for running RISC-V architectural compliance tests (RISCOF) using the new riscv_transpiler VM instead of the legacy risc_v_simulator. Changes: - Add riscv_transpiler/src/riscof.rs with self-contained RISCOF execution - Execute binaries at configurable entry points (default 0x0100_0000) - Extract begin_signature/end_signature symbols from ELF - Write signature files in RISCOF format (hex words, one per line) - Add CLI run-for-riscof command with required --cycles argument - Pin nightly toolchain to 2026-01-10 for reproducible builds - Update feature flags for nightly compatibility (add const_cmp) - Follow codebase idioms: panic/expect instead of anyhow, as_chunks Test results: 42/47 RISCOF tests pass (expected failures: fence, div, rem, mulh, mulhsu - these instructions are not implemented in the zkVM) Co-Authored-By: Claude Opus 4.5 <[email protected]>
2 participants
What ❔
The RISCOF framework is used for differential testing of RISC-V implementations against reference emulators such as the formally specified Sail model. The dashboard here tracks compliance using the canonical RISC-V architecture tests, executed via RISCOF, and in time we may add additional tests as part of a custom test suite.
In the RISCOF approach, the target implementation and the reference both dump a region of memory to a file (the "signatures") for a direct value-by-value comparison. This pull request adds the signature extraction mechanism for Airbender.
Airbender is compliant for every rv32im opcode except
fence. This indicates a very high degree of compliance. For instance, theaddtest suite alone addresses over 500 edge cases. As you know, one can get by without this instruction, but it does complicate the use of certain tools such as fuzzers that may emit such instructions, since they are technically required for compliance.Why ❔
It seems you used to have the same (at least at a glance) tests set up, but the job as been disabled
zksync-airbender/Cargo.toml
Line 19 in 9ed7929
If this PR is accepted, the dashboard will be updated to run a nightly job looking for updates to the main branch, and running the tests in the case of a new commit. This is useful for catching regressions, and also for tracking any ISA changes that may occur.
Is this a breaking change?
Checklist