bin-smoke on the macOS runner: projection_shape_bench fails the job when every tuner timing is refused for host contention

[lukstafi]

Observed on lukstafi/ocannl-staging PR #605 (a workflow-only change), macOS main leg, run https://github.com/lukstafi/ocannl-staging/actions/runs/33679084275/job/100410996322:

(cd _build/default/bin && ./projection_shape_bench.exe 2 2 smoke fwd seeds --ocannl_backend=cc)
   !! smoke_2x2 / default (untuned): every queued tuner timing was refused for host contention
   smoke_2x2: no seed arm was measured
1 cell(s) of this experiment could not be measured or failed parity -- see the !! lines.

The bench exits 1 on an unmeasured cell by design (bin/projection_shape_bench.ml ~line 892: "any cell that could not be measured is a failure of the run, not a blank"). That is the right contract for a benchmark run, but under @bin-smoke the executable is a runtime canary (gh-858), run by dune concurrently with @runtest in the same invocation on a 3-core macOS runner, so host contention is the expected state rather than an anomaly. The log shows test/training/data_parallel.exe running at the same moment.

None of the previous macOS main reds in the last 40 master runs had this cause (they were a tinygrad cache probe, a materialize-flip claim, and an exemption-count drift), so this is a new flake shape rather than a recurrence.

Possible directions, not decided:

• have the smoke rule pass a flag (or env) under which contention refusal on the smoke group is reported but not fatal, keeping the exit-1 contract for real runs;
• or make @bin-smoke depend on @runtest so the canaries run after the suite, not beside it;
• or accept the run-level (locks ...) cost and take the training lock for the timing canaries.

Related: #855, #888, #892 (contention detection in autotune timing), #743 (hermetic driver for bin/ benches), #858 (why bin-smoke exists).

[lukstafi]

Data point: re-running only the failed macOS main job of run 33679084275 passed on the first retry, so the refusal is transient contention on the shared runner, not a deterministic property of the smoke geometry. The PR merged on the green re-run.

[lukstafi]

A data point for the decision here, from lukstafi#653, which added a second macOS member to @bin-smoke: metal_queue_probe.exe --iterations=1024 --deadline-seconds=120.

It is contention-immune by construction rather than by tolerance, which is a fourth direction beside the three listed above -- change what the canary CLAIMS rather than when it runs:

• exact iteration count, so no calibration and no timing target to miss;
• the arm ratios it prints are output, not assertions -- nothing in the run compares them to a threshold;
• the only nonzero exits are a failed command buffer, a stalled event wait, and the SIGALRM deadline.

On the shared macOS runner it took about 50 ms wall (achieved: 3.999 ms single kernel), concurrently with the rest of the walk, and the same run's numbers on an idle M4 Max differ by roughly 4x -- which is exactly the point: nothing in the canary reads them.

projection_shape_bench is harder because the thing it exercises IS a tuner, so "measured nothing" is a real outcome for it in a way it is not for a submission-shape probe. But the framing generalizes: under @bin-smoke the question is whether the pipeline ran end to end, and a bench whose exit code answers a measurement-validity question instead is being asked something the alias never wanted. --allow-unmeasured (already passed there) is that separation done at the call site; a smoke MODE inside the bench, where refusal is reported and not fatal while the exit-1 contract stands for real runs, is the same idea one level in.

[lukstafi]

Closing on the stated exit condition.

lukstafi#608 (merged 2026-09-03, commit 8ebd825 "A refused timing window does not fail the bin-smoke canary") gave projection_shape_bench the --allow-unmeasured flag and bin/dune's @bin-smoke rule passes it, so a contention refusal on the shared runner counts as unmeasured rather than failing the canary, while the exit-1 contract stands for real runs.

Evidence that it holds on the runner it was filed against: the 2026-09-06 scheduled run https://github.com/lukstafi/ocannl-staging/actions/runs/34008717573 (master@df447ccc) has both macOS legs green — Build (macos-latest, 5.5.x, main) and Build (macos-latest, 5.5.x, train) — alongside green ubuntu 5.5.x/5.3.x legs; only the two Windows legs failed, on #935's cygwin pkgconf fetch, which is unrelated. PR #669's run on 2026-09-08 (https://github.com/lukstafi/ocannl-staging/actions/runs/34258301204) repeats the green macOS main leg.

The pattern for future smoke members is the one the 2026-09-05 comment records from staging#653: a canary contention-immune by construction (exact iterations, ratios printed rather than asserted), so the alias asks only whether the pipeline ran end to end.