Cloud parameter sweeps: 25-knob routing experiments overnight for ~$20

[drandyhaas]

Routing quality depends on ~15 tunable defaults (soft proximity costs, A* heuristic weight, via/turn costs, rip-up policy, fanout geometry) that had mostly been inherited rather than measured. Measuring them honestly requires replaying full routing chains across a large board corpus — about 8 CPU-hours per candidate value, or ~5 days of a workstation for one campaign. We built a cloud harness that does it for ~$1 per candidate, ran two campaigns (#584, #586), and shipped four holdout-validated default improvements.

The harness (tests/stress/modal_sweep/)

Every recorded stress-corpus board replays deterministically from its command manifest — no LLM involved, pure CPU. The harness fans this out on Modal:

• One task = one (arm, board): a candidate parameter set ("arm") × one board's full chain replay. 3,000+ tasks run on ~100 rented containers; the wall-clock floor is the single longest board, not the campaign size.
• Arms are JSON: env knobs, routing_defaults.py constants (patched per-container), or manifest rewrites for baked CLI flags.
• Images are pinned: built from a clean git commit with exactly-pinned numeric deps, because we measured two image builds days apart routing 28/96 boards differently at identical settings (numpy/scipy float drift). Every result row records the commit and package versions that produced it; only same-image comparisons are valid, and every wave carries its own baseline arm.
• Grading uses the same DRC/connectivity checkers as local work: the primary metric is the pad-level disconnection count, paired per board against the baseline arm (the router is deterministic, so paired comparisons have no run-to-run noise). Wall time is recorded but load-contaminated; CPU-seconds (getrusage) and A* iteration counts are the honest cost metrics.
• Results bank to a volume as tasks finish; a stopped or crashed campaign resumes by skipping banked rows.

The methodology

A four-stage funnel, each stage only spending on what the cheaper stage couldn't rule out:

1. Prescreen (local, 2 challenging boards, $0): catches values that blow up quality, CPU (>1.5× default), or memory (>2×).
2. Screen (12 cheap, discriminative boards, ~2¢/arm): kills clearly-worse arms against the baseline.
3. Full corpus (~100 curated boards, ~$1/arm): the ranking evidence. Boards were curated by measured information-per-dollar — a third of the corpus showed zero across-arm spread and was dropped.
4. Holdout (75 boards from sets never used in any tuning decision): the veto. Nothing ships without it.

Two rules the data forced on us: measure combinations, never assume additivity (within-family effects composed every time; cross-family combinations broke twice, once at +400 DRC), and holdout-first (tuning-set wins shrank or reversed on fresh boards every single time).

Results

Shipped (holdout: −45% disconnections at −27% route CPU vs the old defaults, DRC neutral-or-better):

default	old → new	why
heuristic_weight	1.9 → 2.3	dose-response peak; both neighbors worse, 1.7/3.0 screen-killed
via_cost	50 → 75	cleaner and cheaper; composes with the heuristic change
proximity_heuristic_factor	0.02 → 0	redundant once the base heuristic is greedier; −8% CPU
QFN fanout extension	0.1 → 0.05 mm	shorter escape stubs leave routing room

Measured and deliberately not shipped: deeper rip-up (5) and sum-based proximity composition each looked like −30 wins on the tuning set, but together on the holdout they erased the other flips' gains while adding 37% CPU — textbook selection-set overfit (the tuning boards were chosen for knob-sensitivity). They remain retry-tier tools for hard boards, alongside track-proximity 2.0, the composition modes, and crossing-penalty 2000 — each real in specific regimes, none default-worthy.

Validated as already-optimal: the stock soft-cost defaults (stub 0.2, rip-avoidance 0.1, via-proximity ×10) all sit at their local optima on the new core. Vertical attraction is definitively dead (2.3× CPU for negative quality), as are the experimental fanout modes.

Cost: ~$120 of cloud compute for both campaigns including every bring-up mistake; a repeat under the final harness is ~$15. The follow-on ideas (#587–#595) can now each be judged for about a dollar.