Skip to content

Tune crossing failure probability calibration to reduce benchmark MSE#649

Closed
0hmX wants to merge 3 commits intomainfrom
codex/improve-mse-in-unravelsolver-tests
Closed

Tune crossing failure probability calibration to reduce benchmark MSE#649
0hmX wants to merge 3 commits intomainfrom
codex/improve-mse-in-unravelsolver-tests

Conversation

@0hmX
Copy link
Copy Markdown
Contributor

@0hmX 0hmX commented Mar 12, 2026

Motivation

  • Reduce Mean Squared Error (MSE) of the high-density benchmark predictor by improving the failure probability model used in routing scoring.
  • Keep changes limited to the existing crossing probability calculation and avoid touching other solver logic or data flows.

Description

  • Updated calculateNodeProbabilityOfFailure in lib/solvers/UnravelSolver/calculateCrossingProbabilityOfFailure.ts to replace the previous via-estimate/capacity ratio with a calibrated model.
  • Added a safeCapacity floor via Math.max(totalCapacity, 0.05) to avoid unstable ratios on very small nodes.
  • Compute a weightedCrossingLoad with adjusted weights, apply layerRelief for multi-layer nodes, convert to utilization against capacity, then map through a logistic calibration and a small baseline floor, clamped to [0,1].
  • Preserved existing hard rules: nodes with _containsTarget return 0 and single-layer nodes with any crossing still return 1.

Testing

  • Ran the high-density benchmark pipeline using the worker harness (runBenchmarkWithWorkers) on representative subsets to measure MSE before and after the change via the scripts/highdensity-benchmark tooling.
  • Baseline MSE (sampled subset): 0.003227983353163425; New MSE (same sampled subset after change): 0.0027581947657854496; absolute improvement 0.0004697885873779752 and relative improvement 14.55% lower MSE.
  • Also verified during development on an additional subset (first 200 cases) where a baseline run recorded 0.0037090839563132883 for reference; all benchmark runs completed successfully using runBenchmarkWithWorkers.
  • Commands used for measurement included running the benchmark harness via bun -e '...runBenchmarkWithWorkers...' and computing MSE with scripts/highdensity-benchmark/metrics/calculateMse.ts, and all automated runs succeeded.

Codex Task

@vercel
Copy link
Copy Markdown

vercel bot commented Mar 12, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
capacity-node-autorouter Ready Ready Preview, Comment Mar 12, 2026 9:00pm

Request Review

@tscircuitbot
Copy link
Copy Markdown

🏃 Benchmark This PR

Run benchmarks by commenting on this PR:

/benchmark [solver-name|all] [scenario-limit] --concurrency <n> --effort <n>

Examples:

  • /benchmark -> AutoroutingPipelineSolver, all scenarios (default concurrency uses the benchmark runner CPU count)
  • /benchmark AutoroutingPipelineSolver -> one solver, all scenarios
  • /benchmark all 20 -> all solvers, first 20 scenarios
  • /benchmark AutoroutingPipelineSolver 20 --concurrency 8 -> one solver, 20 scenarios, 8 workers
  • /benchmark AutoroutingPipelineSolver 20 --effort 2 -> one solver, 20 scenarios, 2x effort

Any PR whose title contains [BENCHMARK TEST] will automatically run the benchmark workflow on PR updates.

@0hmX
Copy link
Copy Markdown
Contributor Author

0hmX commented Mar 12, 2026

/benchmark

@tscircuitbot
Copy link
Copy Markdown

tscircuitbot commented Mar 12, 2026

🏃 Autorouting Benchmark Results

📊 PR Results
Benchmark Results (1x effort)

+---------------------------+----------------+--------------------+-----------+----------+----------+
| Solver                    | Completed %    | Relaxed DRC Pass % | Timed Out | P50 Time | P95 Time |
+---------------------------+----------------+--------------------+-----------+----------+----------+
| AutoroutingPipelineSolver | 78.1% (🕒1.9%) | 1.9% (🕒1.9%)      | 2/105     | 6.7s     | 43.3s    |
+---------------------------+----------------+--------------------+-----------+----------+----------+

Scenarios: 105
📊 Main Branch Results
Benchmark Results (1x effort)

+---------------------------+----------------+--------------------+-----------+----------+----------+
| Solver                    | Completed %    | Relaxed DRC Pass % | Timed Out | P50 Time | P95 Time |
+---------------------------+----------------+--------------------+-----------+----------+----------+
| AutoroutingPipelineSolver | 82.9% (🕒3.8%) | 1.9% (🕒3.8%)      | 4/105     | 6.1s     | 62.9s    |
+---------------------------+----------------+--------------------+-----------+----------+----------+

Scenarios: 105

📦 Artifact: https://github.com/tscircuit/tscircuit-autorouter/actions/runs/23000129686

@0hmX
Copy link
Copy Markdown
Contributor Author

0hmX commented Mar 12, 2026

/benchmark

@tscircuitbot
Copy link
Copy Markdown

tscircuitbot commented Mar 12, 2026

🏃 Autorouting Benchmark Results

📊 PR Results
Benchmark Results (1x effort)

+---------------------------+----------------+--------------------+-----------+----------+----------+
| Solver                    | Completed %    | Relaxed DRC Pass % | Timed Out | P50 Time | P95 Time |
+---------------------------+----------------+--------------------+-----------+----------+----------+
| AutoroutingPipelineSolver | 78.1% (🕒1.0%) | 1.0% (🕒1.0%)      | 1/105     | 5.3s     | 57.3s    |
+---------------------------+----------------+--------------------+-----------+----------+----------+

Scenarios: 105
📊 Main Branch Results
Benchmark Results (1x effort)

+---------------------------+----------------+--------------------+-----------+----------+----------+
| Solver                    | Completed %    | Relaxed DRC Pass % | Timed Out | P50 Time | P95 Time |
+---------------------------+----------------+--------------------+-----------+----------+----------+
| AutoroutingPipelineSolver | 82.9% (🕒3.8%) | 1.9% (🕒3.8%)      | 4/105     | 6.1s     | 62.9s    |
+---------------------------+----------------+--------------------+-----------+----------+----------+

Scenarios: 105

📦 Artifact: https://github.com/tscircuit/tscircuit-autorouter/actions/runs/23001405060

@0hmX
Copy link
Copy Markdown
Contributor Author

0hmX commented Mar 12, 2026

/benchmark

@tscircuitbot
Copy link
Copy Markdown

tscircuitbot commented Mar 12, 2026

🏃 Autorouting Benchmark Results

📊 PR Results
Benchmark Results (1x effort)

+---------------------------+----------------+--------------------+-----------+----------+----------+
| Solver                    | Completed %    | Relaxed DRC Pass % | Timed Out | P50 Time | P95 Time |
+---------------------------+----------------+--------------------+-----------+----------+----------+
| AutoroutingPipelineSolver | 78.1% (🕒1.9%) | 1.9% (🕒1.9%)      | 2/105     | 5.4s     | 33.8s    |
+---------------------------+----------------+--------------------+-----------+----------+----------+

Scenarios: 105
📊 Main Branch Results
Benchmark Results (1x effort)

+---------------------------+-------------+--------------------+-----------+----------+----------+
| Solver                    | Completed % | Relaxed DRC Pass % | Timed Out | P50 Time | P95 Time |
+---------------------------+-------------+--------------------+-----------+----------+----------+
| AutoroutingPipelineSolver | 83.8%       | 1.9%               | 0/105     | 9.5s     | 71.8s    |
+---------------------------+-------------+--------------------+-----------+----------+----------+

Scenarios: 105

📦 Artifact: https://github.com/tscircuit/tscircuit-autorouter/actions/runs/23023603112

@github-actions
Copy link
Copy Markdown

This PR has been automatically marked as stale because it has had no recent activity. It will be closed if no further activity occurs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants