ledger: warm + memory-floor reds resolved as machine swap exhaustion … #584
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: bench | |
| # Competitive-regression gate (see CONTRIBUTING.md "Performance & size invariant"). | |
| # Runs on every push and PR. HARD-GATED here: checksums/parity, wasm sizes, and | |
| # compile success. TIMING ratios (speed pins, geomeans, fastest-wasm, examples, | |
| # floatbeat, perf-fuzz) print informational on CI — a shared 2-core runner reads | |
| # identical builds up to 15× slower, so timing is asserted off-CI where the | |
| # release discipline runs `npm run test:bench` on stable hardware (test/bench.js | |
| # okTiming). | |
| on: | |
| push: | |
| paths-ignore: | |
| - '**/*.md' | |
| - '**/*.svg' | |
| - '**/*.html' | |
| - '.work/**' | |
| - 'LICENSE' | |
| - 'CONTRIBUTING.md' | |
| - 'bench/results.json' | |
| - 'bench/results-ci.json' | |
| pull_request: | |
| paths-ignore: | |
| - '**/*.md' | |
| - '**/*.svg' | |
| - '**/*.html' | |
| - '.work/**' | |
| - 'LICENSE' | |
| - 'CONTRIBUTING.md' | |
| - 'bench/results.json' | |
| - 'bench/results-ci.json' | |
| jobs: | |
| bench: | |
| runs-on: ubuntu-latest | |
| # The publish step commits bench/results-ci.json back to main; | |
| # the default GITHUB_TOKEN is read-only and 403s that push without this. | |
| permissions: | |
| contents: write | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: actions/setup-node@v4 | |
| with: | |
| node-version: 22 | |
| - run: npm install | |
| # Zig native reference rows (cases with a .zig baseline). | |
| - uses: mlugg/setup-zig@v2 | |
| with: | |
| version: 0.13.0 | |
| # Extra "Run as JS" runtimes (Deno, Bun). bench.mjs gates every target on | |
| # `available`, so a missing tool just drops its row — additive, never fails. | |
| - uses: denoland/setup-deno@v2 | |
| with: | |
| deno-version: v2.x | |
| - uses: oven-sh/setup-bun@v2 | |
| # Safari's engine: standalone JavaScriptCore shell via jsvu (→ ~/.jsvu/bin/jsc). | |
| # Non-fatal — bench.mjs drops the jsc row if the install fails. | |
| - name: install jsc (JavaScriptCore) | |
| run: | | |
| npx --yes jsvu --engines=javascriptcore --os=linux64 || true | |
| echo "$HOME/.jsvu/bin" >> "$GITHUB_PATH" | |
| # NumPy native-reference rows (python3 is preinstalled on ubuntu). | |
| - name: install numpy | |
| run: python3 -m pip install --break-system-packages numpy || pip install numpy || true | |
| # GNU time provides /usr/bin/time — bench.mjs wraps every measured run in it | |
| # to read peak RSS (the memKb column). Non-fatal: absent, memKb stays null. | |
| - name: install GNU time | |
| run: sudo apt-get install -y time || true | |
| # wasmtime runs the standalone `jz → wasmtime` / `jz → wasm2c` reference rows. | |
| - name: install wasmtime | |
| run: | | |
| curl https://wasmtime.dev/install.sh -sSf | bash || true | |
| echo "$HOME/.wasmtime/bin" >> "$GITHUB_PATH" | |
| # Pin zig: the runner's preinstalled zig (0.13) fails every bench zig/ | |
| # zig-wasm build (API churn) — 0/43 rows made those rivals silently | |
| # uncontested until the coverage gate hardened. Match the version the | |
| # sources build with locally. | |
| - name: pin zig | |
| uses: mlugg/setup-zig@v2 | |
| with: | |
| version: 0.16.0 | |
| # TinyGo (the lean go→wasm rival — a separate row from go-wasm's stock | |
| # toolchain). Non-fatal: the target self-gates on availability. | |
| # TinyGo must MATCH the runner's Go: 0.35 caps at go1.23 and the runner | |
| # ships go1.24, which produced 43 silent FAIL rows on the first hard-gate | |
| # run. Pin a release that accepts the installed Go, and smoke-build | |
| # wasip1 so a version skew fails THIS step loudly instead of degrading | |
| # into an absent rival (the required-rivals gate in test/bench.js reds | |
| # when a mandatory producer is missing). | |
| - name: install tinygo | |
| run: | | |
| curl -fsSL -o /tmp/tinygo.deb https://github.com/tinygo-org/tinygo/releases/download/v0.41.1/tinygo_0.41.1_amd64.deb | |
| sudo dpkg -i /tmp/tinygo.deb | |
| tinygo version | |
| printf 'package main\nfunc main() {}\n' > /tmp/tg.go | |
| tinygo build -target=wasip1 -o /tmp/tg.wasm /tmp/tg.go | |
| # Rust→wasm baseline needs the wasm target added to the preinstalled rustc; | |
| # the runner ships rustc but NOT the wasm32-wasip1 std, so without this every | |
| # rust-wasm case fails ("can't find crate for std") and the hero's Rust→wasm | |
| # stat goes null (why the 3rd hero stat was dropped in 83d6add5). | |
| - name: rust wasm target | |
| run: rustup target add wasm32-wasip1 || true | |
| # Competitor toolchains + binaryen (provides wasm-opt). | |
| - run: npm install --no-save assemblyscript@~0.27.0 binaryen | |
| # Porffor: the 2026 rewrite (fixes typed-array OOB, module-scope refs, JSON - bench | |
| # coverage 13/52 -> full sweep) lives on git main only (npm 0.61.x is the old engine, | |
| # and the restructured repo is not npm-installable yet). Pin a known-good commit | |
| # (2026-07-24 head — full-corpus sweep validated on the reference machine, 42 | |
| # parity-ok + porf-native lane); bench.mjs probes the CLI shape via --version, | |
| # so either vintage works. | |
| - name: porffor (git, pinned) | |
| run: | | |
| git clone https://github.com/CanadaHonk/porffor.git "$RUNNER_TEMP/porffor" || true | |
| git -C "$RUNNER_TEMP/porffor" checkout 8ef8aaaad9199de14a35bd1aa214f4d3e23bf97a || true | |
| echo "PORF_BIN=$RUNNER_TEMP/porffor/porf" >> "$GITHUB_ENV" | |
| - name: expose toolchains on PATH | |
| run: echo "$PWD/node_modules/.bin" >> "$GITHUB_PATH" | |
| - name: versions | |
| run: | | |
| node --version | |
| asc --version || true | |
| "$PORF_BIN" --version || true | |
| wasm-opt --version || true | |
| - run: npm run test:bench | |
| # ── native-lane smoke gate (audit-#12 step 4) ────────────────────────── | |
| # jz-w2c/jz-w2c2 are NOT installed for the main test:bench run above (kept | |
| # scoped to this step's own env — the full corpus stays on the reference | |
| # machine's toolchain, unchanged runtime/risk for every other CI run). | |
| # This proves both native translators still (a) build, (b) run, (c) match | |
| # jz's own checksum on a few representative cases — the regression a | |
| # translator-side codegen bug (like audit-#12's wasm2c `return_call` | |
| # opcode-0x12 defect) would trip. nqueens is that exact repro case, kept | |
| # in the smoke set as a permanent regression guard for it. | |
| - name: install wasm2c (wabt prebuilt) | |
| run: | | |
| curl -sL -o "$RUNNER_TEMP/wabt.tar.gz" https://github.com/WebAssembly/wabt/releases/download/1.0.41/wabt-1.0.41-linux-x64.tar.gz | |
| tar xzf "$RUNNER_TEMP/wabt.tar.gz" -C "$RUNNER_TEMP" | |
| "$RUNNER_TEMP"/wabt-1.0.41/bin/wasm2c --version | |
| - name: install w2c2 (build from source — no prebuilt/package exists) | |
| run: | | |
| git clone --depth 1 https://github.com/turbolent/w2c2.git "$RUNNER_TEMP/w2c2" | |
| cmake -B "$RUNNER_TEMP/w2c2/w2c2/build" "$RUNNER_TEMP/w2c2/w2c2" | |
| cmake --build "$RUNNER_TEMP/w2c2/w2c2/build" --target w2c2 -j | |
| "$RUNNER_TEMP/w2c2/w2c2/build/w2c2" -h | |
| - name: native-lane smoke (jz-w2c + jz-w2c2, checksum-verified) | |
| env: | |
| WABT_W2C_DIR: ${{ runner.temp }}/wabt-1.0.41/share/wabt/wasm2c | |
| W2C2_DIR: ${{ runner.temp }}/w2c2/w2c2 | |
| W2C2_BIN: ${{ runner.temp }}/w2c2/w2c2/build/w2c2 | |
| run: | | |
| export PATH="$RUNNER_TEMP/wabt-1.0.41/bin:$PATH" | |
| node bench/bench.mjs --targets=jz-w2c,jz-w2c2 --cases=nqueens,tokenizer,wordcount --json="$RUNNER_TEMP/native-smoke.json" || exit 1 | |
| node -e ' | |
| const r = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/native-smoke.json", "utf8")); | |
| let bad = []; | |
| for (const [id, c] of Object.entries(r.cases)) { | |
| for (const tid of ["jz-w2c", "jz-w2c2"]) { | |
| const t = c.targets?.[tid]; | |
| if (!t) { bad.push(`${id}/${tid}: missing`); continue; } | |
| if (t.status === "fail") { bad.push(`${id}/${tid}: FAIL — ${t.reason}`); continue; } | |
| if (t.parity !== "ok") bad.push(`${id}/${tid}: parity=${t.parity}`); | |
| } | |
| } | |
| if (bad.length) { console.error("native-lane smoke FAILED:\n" + bad.join("\n")); process.exit(1); } | |
| console.log("native-lane smoke OK — jz-w2c + jz-w2c2 both build/run/checksum-match on", Object.keys(r.cases).length, "cases"); | |
| ' | |
| # Refresh the bench-page dataset (results.json + bench.svg). Push-to-main | |
| # only; the paths-ignore above keeps the commit from re-triggering this | |
| # workflow. The page's live in-browser runner loads bench/web/*.wasm, which | |
| # is NOT committed: pages.yml rebuilds it from source at deploy time | |
| # (node bench/bench.mjs --emit-web) — same model as dist/ and example wasm. | |
| # clang/rustc/go/zig are preinstalled on ubuntu runners, so every | |
| # rival builds on the one consistent machine. The headline axis is wasm-in-V8 | |
| # (c-wasm/rust-wasm/go-wasm/zig-wasm — apples-to-apples with jz); native | |
| # C/Rust/Go/Zig stay as a corpus reference band (the aggregate "what you give | |
| # up by not rewriting" ceiling). Per case the page splits into a WASM lane and | |
| # a Native lane (jz-w2c = jz lowered to native via wasm2c → clang, vs native | |
| # toolchains). jz-w2c self-gates on wabt's wasm2c runtime + SIMDe headers | |
| # (WABT_W2C_DIR / SIMDE_DIR); absent here, so it skips and the page's Native | |
| # lane stays hidden until a wasm2c toolchain is wired into CI. Every target | |
| # that runs lands in the dataset; index.html decides which lane shows it. | |
| - name: publish bench snapshot | |
| if: github.event_name == 'push' && github.ref == 'refs/heads/main' | |
| run: | | |
| node bench/bench.mjs --targets=nat,c-wasm,rust-wasm,go-wasm,tinygo,zig-wasm,rust,zig,go,v8,deno,bun,jsc,as,porf-native,jz,jz-wasmtime,jz-w2c,wat,numpy --json || exit 1 | |
| git config user.name 'github-actions[bot]' | |
| git config user.email 'github-actions[bot]@users.noreply.github.com' | |
| # PER-MACHINE datasets: bench/results.json is the REFERENCE-machine | |
| # evidence (Apple M4, committed by the release discipline — the strict | |
| # fastest-wasm claim and the jz-w2c native lane are scoped to it; this | |
| # runner has no wasm2c toolchain and different V8 tiering, so letting | |
| # it overwrite the reference file both dropped the native lane and | |
| # made the headline numbers machine-dependent). The runner's own | |
| # measurement is published as the SECONDARY dataset results-ci.json. | |
| mv bench/results.json bench/results-ci.json | |
| git checkout -- bench/results.json bench/bench.svg | |
| git add bench/results-ci.json | |
| git diff --cached --quiet || git commit -m 'bench: refresh results-ci.json (runner dataset) [skip ci]' | |
| # Two waves pushed minutes apart both refresh the same GENERATED files — | |
| # a plain rebase conflicts on the SVG and kills the run. `-X theirs` | |
| # (in rebase terms: the commit being replayed, i.e. OUR snapshot) auto- | |
| # resolves. Retry covers a push landing mid-rebase. STALE GUARD: the | |
| # dataset measured $GITHUB_SHA — if main's newest SOURCE commit (skip | |
| # generated snapshot commits) has moved past it, a late-finishing run | |
| # would overwrite fresher evidence with older; skip and let the newer | |
| # push's own bench run publish. | |
| for i in 1 2 3; do | |
| git fetch origin main | |
| latest_src=$(git log origin/main --format='%H %s' -n 50 | grep -v 'bench: refresh' | head -1 | cut -d' ' -f1) | |
| if [ -n "$latest_src" ] && [ "$latest_src" != "$GITHUB_SHA" ]; then | |
| echo "stale snapshot: measured $GITHUB_SHA, main source tip is $latest_src — skipping publish" | |
| exit 0 | |
| fi | |
| git push && break | |
| git pull --rebase -X theirs origin main || { git rebase --abort || true; exit 1; } | |
| done |