[Evaluation] Add versioned offline confidence calibration artifacts

[yuki-uix]

Related #3475

Purpose

This PR implements the offline confidence-calibration artifact workflow for accepted issue #3475.

It adds:

• deterministic train, calibration, and held-out dataset splits;
• paired small-model and large-model confidence collection;
• Brier score, ECE, accuracy, escalation, uplift, regression, and coverage metrics;
• threshold selection using calibration data only;
• held-out evaluation and failure-slice reporting;
• versioned manifests, source hashes, model metadata, rollback guidance, and candidate configuration diff;
• checked-in fixtures for deterministic offline tests.

Affected modules:

• tools/agent/scripts/tuning/
• tools/agent/scripts/tuning/verify_results/

This PR does not modify runtime routing policy, automatically promote thresholds, train models, or commit provider-specific live API results.

Test Plan

• PYTHONPATH=tools/agent/scripts python -m pytest tools/agent/scripts/tuning/tests -q
• make agent-ci-lint AGENT_BASE_REF=upstream/main AGENT_CHANGED_FILES_PATH=/tmp/semantic-router-changed-files.txt
• Prepared and collected a real MMLU-Pro evaluation dataset.
• Rebuilt the artifact twice and verified deterministic artifact IDs.

Test Result

• Tuning tests: 60 passed.
• Agent validation completed successfully.
• Manifest and workflow contract validation passed.
• Repository validation tests passed.
• Formatting, Ruff, AST supply-chain scan, and changed-file checks passed.
• Real-data collection completed for 350 questions:
  • train: 210
  • calibration: 70
  • held-out: 70
• Candidate threshold: 0.9683.
• Held-out evaluation showed no net uplift and one regression.
• The artifact remains candidate; no production threshold was promoted.
• No runtime policy was changed.

Local live artifacts

I generated a 350-question calibration artifact locally using DashScope-hosted Qwen models. I intentionally did not commit the provider-specific live results by default because they depend on a particular API provider, model snapshot, endpoint configuration, dataset revision, and sampling run. Committing them could make the results appear to be a canonical benchmark baseline, while they are primarily evidence that the workflow works end to end.

The PR therefore commits deterministic fixtures for CI and keeps the live artifact as local validation evidence. The locally generated artifact is reproducible from the documented workflow and has been verified for deterministic rebuilding.

If maintainers prefer to keep the full live artifact in the repository as a versioned reference, I am happy to add it with its manifest, model/provider metadata, dataset hash, generation configuration, and artifact ID.

---

Semantic Router PR Checklist

• PR title begins with exactly one bracketed category
• The title does not stack prefixes
• The PR links an accepted issue with exactly one recognized owner
• Commits in this PR are signed off with git commit -s
• The Purpose, Test Plan, and Test Result sections reflect the actual scope, commands, and blockers

See CONTRIBUTING.md for the full contributor workflow and commit guidance.

[netlify]

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	19de525
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/6a9f6576c77dd000081fb606
😎 Deploy Preview	https://deploy-preview-3541--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Xunzhuo]

Thanks for putting the workflow and provenance surfaces together. The threshold sweep still misses valid operating points: OfflineAnalyzer only probes confidence plus 0.001, so two calibration scores less than 0.001 apart cannot be separated. For example, an uplift at 0.5000 and a regression at 0.5005 have a safe threshold between them, but the candidates jump from no escalation to escalating both. That can select a worse threshold or report no safe threshold even when one exists. Please enumerate exact score boundaries, for example with the next representable float or adjacent midpoints, and add a close-score regression.

[yuki-uix]

Thanks for catching this edge case. You’re right that using a fixed +0.001 margin can skip a valid operating point when adjacent confidence scores are closer than 0.001.

I pushed a fix that:

• generates threshold candidates from the policy regions between adjacent observed confidence scores;
• uses midpoints, with math.nextafter as a floating-point fallback;
• preserves full precision during threshold evaluation;
• adds a regression test covering an uplift at 0.5000 and a regression at 0.5005;
• documents the boundary-based candidate selection.

Validation completed:

• 61 passed;
• the agent validation gate passed;
• the real artifact was rebuilt twice with the same artifact ID:
  sha256:72b15ea4c605331a9fd12059fc3f82c3884be716cc2857e13b046d025a8a2012;
• the artifact remains candidate, and no runtime policy was changed.