Skip to content

ci: move bench workflow off sunset buildjet runner#706

Merged
Jon-Becker merged 1 commit into
mainfrom
chore/bench-github-runner
Jul 5, 2026
Merged

ci: move bench workflow off sunset buildjet runner#706
Jon-Becker merged 1 commit into
mainfrom
chore/bench-github-runner

Conversation

@Jon-Becker

@Jon-Becker Jon-Becker commented Jul 5, 2026

Copy link
Copy Markdown
Owner

What changed? Why?

Buildjet has been sunset and can no longer schedule its buildjet-16vcpu-ubuntu-2204 runner. This PR migrates the bench workflow to GitHub's standard ubuntu-latest hosted runner to keep CI unblocked.

Changed: .github/workflows/bench.yml — replaced runs-on: buildjet-16vcpu-ubuntu-2204 with runs-on: ubuntu-latest

Notes to reviewers

  • Only one line changed in .github/workflows/bench.yml
  • All other workflow logic is preserved: still triggers on pull_request, maintains the same concurrency group with cancel-in-progress, uses boa-dev/criterion-compare-action@v3, and keeps the old-benchmark-comment cleanup script
  • ubuntu-latest is a 4vCPU runner (vs the original 16vCPU buildjet box). If higher core count is needed, we can upgrade to a GitHub larger runner (e.g. ubuntu-latest-16-cores), but that requires provisioning in repo/org settings

How has it been tested?

  • Verified the workflow syntax is valid
  • The action configuration and all steps remain unchanged and will work with the new runner

- replace buildjet-16vcpu-ubuntu-2204 with github-hosted ubuntu-latest
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

❌ Eval Report for 79e5905

Test Case CFG Decompilation
TransientStorage 100 30
Mapping 100 95
NestedMappings 100 35
NestedMapping 100 10
Events 100 62
SimpleStorage 100 88
SimpleLoop 85 10
WhileLoop 72 10
NestedLoop 65 15
WETH9 100 67
Average 92 42
⚠️ 8 eval(s) scoring <70%

TransientStorage (CFG: 100, Decompilation: 30)

Decompilation

{
  "score": 30,
  "summary": "The decompilation fails to preserve the logic of five of the six functions. Only setTempOwner is recovered as a real function with a recognizable transient-storage write; incrementCounter, lock, unlock, getCounter, and isLocked are collapsed into meaningless constant declarations that discard their control flow, storage operations, and return logic. The use of TLOAD/TSTORE (transient storage) is only partially reflected via the transient[] mapping in the one recovered function.",
  "differences": [
    "incrementCounter is represented as an empty 'bytes public constant incrementCounter', losing the transient counter read, +1 increment, and write-back entirely.",
    "getCounter is represented as 'uint256 public constant getCounter = 1' instead of a view function returning the transient counter value; the storage read and return are lost.",
    "isLocked is represented as 'bool public constant isLocked = 0xBool(true)' instead of a view function returning the transient locked bool; the storage read is lost.",
    "lock is represented as an empty 'bytes public constant lock', losing the transient write of locked = true.",
    "unlock is represented as an empty 'bytes public constant unlock', losing the transient write of locked = false.",
    "setTempOwner is marked 'pure' though it performs a transient storage write (should be a state-mutating function), and it includes a spurious require(arg0 == address(arg0)) mask check and packed slot arithmetic not present in the source; the core transient write of tempOwner is present but obscured."
  ]
}

NestedMappings (CFG: 100, Decompilation: 35)

Decompilation

{
  "score": 35,
  "summary": "The decompilation fails to preserve the core behavior of the nested-mapping contract. Both read paths (the allowance view function and the public mapping getter) are reduced to empty pure stubs that read no storage and return nothing, and while approve does perform a storage write, it derives the slot from only the spender key, dropping the msg.sender component of the nested mapping. Only a partial fragment of one of three functions survives.",
  "differences": [
    "allowance(address,address) (selector 0xdd62ed3e) is decompiled as an empty pure function that neither reads the nested mapping nor returns a value; the original returns allowances[owner][spender]",
    "The public getter allowances(address,address) (selector 0x55b6ed5c) is decompiled as an empty pure stub, losing the storage read and return entirely",
    "approve computes the storage slot using only the spender argument, omitting the msg.sender key of the nested mapping, so the write targets the wrong slot and does not reflect allowances[msg.sender][spender] = amount",
    "The nested mapping key derivation (double keccak hashing of two keys) is not represented anywhere in the output"
  ]
}

NestedMapping (CFG: 100, Decompilation: 10)

Decompilation

{
  "score": 10,
  "summary": "The decompilation fails to capture the fundamental behavior of the contract. Every original function performs nested-mapping storage reads or writes (SSTORE/SLOAD with hashed slot computation), but the decompiled output contains only pure functions with tautological `require(arg0 == arg0)` / `require(arg0 == address(arg0))` checks and no storage operations whatsoever. None of the setter logic (writing amount/value/bool into the nested mapping slots) or getter logic (returning stored values) is preserved.",
  "differences": [
    "All storage writes are missing: setAllowance, setGrid, and setDeepNested each perform an SSTORE into a nested mapping slot, but the decompiled functions contain no storage assignment.",
    "All storage reads are missing: getAllowance and the public getter functions should SLOAD from nested mapping slots and return a value, but the decompiled functions are marked pure and return nothing.",
    "Nested mapping slot computation (keccak-based key hashing for double/triple nesting) is entirely absent from the output.",
    "Function mutability is wrong: original setters are state-modifying and getters are view, but all decompiled functions are marked pure.",
    "Return values are lost: getAllowance and getter functions should return uint256/bool values; the decompiled functions have no return.",
    "The decompiled functions reduce to no-op tautological requires (arg0 == arg0), representing none of the actual arithmetic or state-change logic."
  ]
}

Events (CFG: 100, Decompilation: 62)

Decompilation

{
  "score": 62,
  "summary": "Most event-emitting functions are preserved with the correct event and arguments (Deposit, Withdrawal, LogBytes, Log, and the multi-emit function combining Deposit/Transfer/Log). However, two of the seven functions lost their emissions entirely: emitTransfer and emitApproval are decompiled as single-argument functions that only perform an identity require and emit nothing, and the Approval event does not appear anywhere in the output. The Log/LogBytes calldata argument handling is also garbled (var_b expressions), though the emit itself is captured.",
  "differences": [
    "emitTransfer(from,to,value) is decompiled (Unresolved_23de6651 / Unresolved_5687f2b8) as a single-argument function that performs no event emission; the Transfer emission with three parameters is lost.",
    "emitApproval(owner,spender,value) is not represented: the Approval event is absent from the contract entirely and no function emits it.",
    "emit Log and emit LogBytes calldata arguments are represented with malformed/undefined expressions (var_b + 0x20 - var_b, extra length operands) rather than a clean string/bytes argument, though the correct event is emitted.",
    "The emitMultiple Log emission encodes the literal string as a raw word constant plus length rather than a readable string, but the three emissions (Deposit, Transfer with address(0), Log) are otherwise preserved."
  ]
}

SimpleLoop (CFG: 85, Decompilation: 10)

Decompilation

{
  "score": 10,
  "summary": "The decompilation fails to capture the core behavior of the function. The original performs a loop that increments the storage variable `number` on each iteration, mutating state. The decompiled output has no loop, no storage write (it is incorrectly marked `view`), and the increment of `number` is entirely absent. Only the storage variable declaration and the reading of `number` are vaguely reflected in nonsensical require statements.",
  "differences": [
    "The for-loop control flow is completely missing from the decompiled output",
    "The storage write `number++` (increment of the state variable) is not represented; the function is incorrectly marked `view`, indicating no state change was detected",
    "The loop bound check `i < loops` is misrepresented as a nonsensical `require(!0 < arg0)` rather than actual iteration logic",
    "No state mutation occurs in the decompiled version, whereas the original mutates `number` `loops` times"
  ]
}

WhileLoop (CFG: 72, Decompilation: 10)

Decompilation

{
  "score": 10,
  "summary": "The decompilation fails to capture the fundamental behavior of the contract. The original loops `loops` times incrementing the storage variable `number`, but the decompiled output contains no loop, no storage write, and is incorrectly marked as `view`. The core logic (iteration and state mutation) is entirely absent.",
  "differences": [
    "The while loop control flow is completely missing; there is no iteration construct in the decompiled output",
    "The storage write to `number` (number = number + 1) is not preserved; the function performs no state changes",
    "The function is incorrectly marked `view`, whereas the original mutates state and is non-view",
    "The loop counter `i` and its comparison/increment logic are absent",
    "The decompiled body consists of meaningless require assertions (e.g. require(arg0 == arg0), require(!0 < arg0)) that do not correspond to the original logic"
  ]
}

NestedLoop (CFG: 65, Decompilation: 15)

Decompilation

{
  "score": 15,
  "summary": "The decompilation fails to capture the fundamental behavior of the function. The original performs nested loops that repeatedly increment the storage variable `number`, but the decompiled output contains no loop structure, no storage write, and incorrectly marks the function as `view`. Only faint residual hints of the loop bounds and increment survive as meaningless require statements.",
  "differences": [
    "The nested for-loop control flow (outer and inner loops iterating `loops` times each) is entirely absent; there is no iteration in the decompiled output.",
    "The storage write `number += 1` (SSTORE incrementing state) is missing; the decompiled code never modifies storage, only contains a require referencing `number + 1`.",
    "Function mutability is wrong: decompiled as `public view`, but the original mutates state (non-view/non-pure).",
    "The loop body's cumulative side effect (incrementing `number` loops*loops times) is not represented at all."
  ]
}

CFG

{
  "score": 65,
  "summary": "The CFG captures the full function dispatch, both loop-condition heads, the loop body, and all revert paths, but the loop iteration structure is incomplete: the inner loop's exit branch and the normal (non-overflow) back-edge continuations for both loops are not represented, so the nested loops appear largely linear rather than cyclic.",
  "missing_paths": [
    "Inner loop exit branch: node 8 (inner condition `j < loops`) has a JUMPI to 0xb1 for the false case (inner loop finished), but no edge/node represents this exit toward the outer-loop increment.",
    "Inner loop back-edge / body-success continuation: node 9 (body `number += 1`) has a JUMPI to 0x01ac for the non-overflow case (increment j, jump back to inner condition), but the CFG only shows the overflow-panic edge (9->10); the successful loop continuation is absent.",
    "Outer loop back-edge: after the inner loop completes (0xb1), control increments i and jumps back to the outer condition (node 7, 0x77), but this iteration back-edge is not present, so the outer loop is not shown as a cycle."
  ],
  "extra_paths": [
    "CALLVALUE non-payable guard (node 0 -> node 1 revert) - compiler-added.",
    "Calldata size / ABI-decode bounds reverts (nodes 12, 6, 15) - compiler-added.",
    "Arithmetic overflow Panic(0x11) revert path (node 9 -> node 10) - compiler-added overflow check on `number += 1`."
  ],
  "observations": [
    "Both source conditionals (the outer `i < loops` and inner `j < loops` for-loop tests) have their condition-check nodes present (nodes 7 and 8), and the outer loop has both its true (->8) and false/exit (->11) branches.",
    "Function entry points for both `loop()` and the auto-generated `number()` getter are captured, along with their exit points (STOP at node 11, RETURN at node 14).",
    "The core structural gap is loop back-edges: JUMPI targets 0xb1 and 0x01ac are decoded within node bodies but never materialize as graph edges, so the graph does not model repeated iteration. This affects both nested loops."
  ]
}

WETH9 (CFG: 100, Decompilation: 67)

Decompilation

{
  "score": 67,
  "summary": "The decompilation recovers the overall WETH9 structure and preserves the core logic of most functions (deposit, withdraw, totalSupply, transfer, transferFrom) including balance checks, arithmetic on balances, the ETH transfer in withdraw, and all four event emissions with correct indexed parameters. However, the allowance (nested mapping) handling is functionally incorrect in several places, and there is a storage-slot resolution inconsistency plus unreachable duplicated code blocks.",
  "differences": [
    "approve() writes the allowance to the balance mapping (storage_map_c) keyed only by the spender (arg0) instead of the allowance mapping keyed by [msg.sender][spender], so it does not update allowance state correctly.",
    "The nested allowance mapping key computation ([src][msg.sender] / [owner][spender]) is not preserved; both allowance() and transferFrom() index the mapping by a single overwritten key rather than the composite owner+spender key.",
    "In transferFrom(), the conditional allowance decrement (require(allowance[src][msg.sender] >= wad); allowance[src][msg.sender] -= wad;) for the non-owner, non-infinite-allowance case is not represented; the allowance check reads from the balance mapping and compares to uint(-1) but never decrements allowance.",
    "The balanceOf() getter reads from storage_map_d while all balance writes (deposit/withdraw/transfer) target storage_map_c, so the read path resolves to a different slot than the write path.",
    "transfer() and transferFrom() contain duplicated, unreachable statement blocks after the initial return (extra balance subtractions and emits), reflecting imperfect branch reconstruction of the inlined transferFrom logic.",
    "The withdraw() external ETH send is modeled as address(msg.sender).transfer(arg0) returning (success, ret) but the original uses msg.sender.transfer which reverts on failure; the success value is not checked (minor, behavior is roughly equivalent since transfer reverts)."
  ]
}

@Jon-Becker Jon-Becker merged commit b1d466b into main Jul 5, 2026
14 of 15 checks passed
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

✅ Coverage Report for 79e5905

Metric Value
Base branch 66.26%
PR branch 66.26%
Diff +0.01%

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Benchmark for 79e5905

Click to view benchmark
Test Base PR %
heimdall_cfg/complex 13.2±0.36ms 13.5±0.40ms +2.27%
heimdall_cfg/simple 1355.9±10.96µs 1380.4±47.92µs +1.81%
heimdall_decoder/seaport 57.3±2.79µs 56.9±2.68µs -0.70%
heimdall_decoder/transfer 4.1±0.26µs 4.1±0.34µs 0.00%
heimdall_decoder/uniswap 15.8±0.69µs 15.8±0.93µs 0.00%
heimdall_decompiler/abi_complex 53.4±0.65ms 54.7±0.87ms +2.43%
heimdall_decompiler/abi_simple 1438.1±46.29µs 1434.9±14.17µs -0.22%
heimdall_decompiler/sol_complex 75.4±1.14ms 76.6±1.47ms +1.59%
heimdall_decompiler/sol_simple 2.2±0.01ms 2.2±0.01ms 0.00%
heimdall_decompiler/yul_complex 59.8±1.00ms 59.1±1.00ms -1.17%
heimdall_decompiler/yul_simple 1594.2±90.15µs 1577.9±39.41µs -1.02%
heimdall_disassembler/complex 1283.4±53.81µs 1279.1±57.50µs -0.34%
heimdall_disassembler/simple 64.4±7.11µs 63.5±4.83µs -1.40%
heimdall_vm/erc20_transfer 262.1±10.00µs 263.6±11.12µs +0.57%
heimdall_vm/fib 848.6±3.72µs 852.4±33.63µs +0.45%
heimdall_vm/ten_thousand_hashes 1372.1±1786.74ms 757.0±88.48ms -44.83%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant