[Router][CLI][Dashboard][Docs] Support decision-scoped candidate quality by duchangyong · Pull Request #2714 · vllm-project/semantic-router

duchangyong · 2026-07-28T12:52:41Z

Closes #2711

Purpose

Add an optional routing.decisions[].modelRefs[].quality_score override so the same model can carry different quality priors in different decisions.
Preserve an explicit zero, validate finite values in the 0..1 range, and fall back to the model-card quality score only when the decision override is omitted.
Apply the override in multi_factor selection and expose the same contract through the CLI schema, dashboard editor, reference configuration, and documentation.
Affected modules: Router / CLI / Dashboard / Docs / E2E.

Test Plan

Run make agent-ci-gate ENV=cpu.
Run the routing-strategies E2E profile in an isolated Podman/Kind cluster.
Run the CLI container integration suite with Podman.
When the default Prometheus host port is already allocated, rerun runtime lifecycle contracts with the supported VLLM_SR_PORT_OFFSET setting and verify the simulator sidecar environment plus /healthz endpoint on the offset port.

Test Result

make agent-ci-gate ENV=cpu: passed, including lint, reference-config checks, the complete Router test suite, CLI tests, Dashboard checks, E2E build, and documentation validation.
routing-strategies E2E: decision-scoped-multi-factor passed 1/1 in an isolated Podman/Kind cluster, verifying three real chat-completion routing paths: model-card quality fallback selected premium-model, reversed decision-scoped quality overrides selected economy-model, and the cost policy continued to select economy-model (verified_policies=3).
CLI Podman integration suite: 35 of 39 tests passed on the default ports; four runtime-start tests were blocked by an existing host binding on port 9090.
The environment propagation, running-container contract, and stop lifecycle tests all passed after applying a port offset. The simulator sidecar contract was also verified through its injected service URL and successful /healthz response on the offset port.
No known code blocker remains for the affected modules.

Semantic Router PR Checklist

PR title uses module-aligned prefixes such as [Router], [CLI], [Dashboard], [Operator], [Fleet-Sim], [Bindings], [Training], [E2E], [Docs], or [CI/Build]
If the PR spans multiple modules, the title includes all relevant prefixes
Commits in this PR are signed off with git commit -s
The Purpose, Test Plan, and Test Result sections reflect the actual scope, commands, and blockers for this change

Signed-off-by: 0027020836 <du.changyong2@iwhalecloud.com>

netlify · 2026-07-28T12:52:48Z

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	`ebcb0c9`
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/6a6e3ba0e592c100087ee758
😎 Deploy Preview	https://deploy-preview-2714--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

github-actions · 2026-07-28T13:01:13Z

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository:

📁 `config`

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

config/README.md
config/config.yaml

📁 `dashboard`

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

dashboard/frontend/src/pages/ConfigPageDecisionModelQuality.test.tsx
dashboard/frontend/src/pages/ConfigPageDecisionsSection.tsx
dashboard/frontend/src/pages/configPageSupport.ts

📁 `deploy`

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

deploy/kubernetes/routing-strategies/aigw-resources/base-model.yaml

📁 `e2e`

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

e2e/profiles/routing-strategies/values.yaml
e2e/testcases/decision_scoped_multi_factor.go

📁 `src/semantic-router`

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

src/semantic-router/pkg/config/canonical_config.go
src/semantic-router/pkg/config/decision_config.go
src/semantic-router/pkg/config/decision_reference_validation_test.go
src/semantic-router/pkg/config/validator_decision.go
src/semantic-router/pkg/selection/multi_factor.go
src/semantic-router/pkg/selection/multi_factor_test.go

📁 `src/vllm-sr`

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

src/vllm-sr/cli/algorithms.py
src/vllm-sr/tests/test_config_contract.py

📁 `website`

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

website/docs/installation/configuration.md
website/docs/proposals/unified-config-contract-v0-3.md
website/docs/tutorials/algorithm/selection/multi-factor.md

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

github-actions · 2026-07-28T13:16:10Z

✅ Supply Chain Security Report — All Clear

Scanner	Status	Findings
AST Codebase Scan (Py, Go, JS/TS, Rust)	✅	31 finding(s) — MEDIUM: 24 · LOW: 7
AST PR Diff Scan	✅	No issues detected
Regex Fallback Scan	✅	No issues detected

Scanned at 2026-08-01T18:40:31.028Z · View full workflow logs

Xunzhuo · 2026-08-01T03:24:29Z

 		if modelRef.Model == "" {
 			continue
 		}
+		if modelRef.QualityScore != nil && (math.IsNaN(*modelRef.QualityScore) || *modelRef.QualityScore < 0 || *modelRef.QualityScore > 1) {


[P2] Validate quality scores in candidate iterations too. This check only walks Decision.ModelRefs, but ModelRef is also used by candidateIterations[].models. Values such as .nan or 2 therefore pass canonical validation on that path even though the same values are rejected in modelRefs. Please reuse the quality-score validation from validateDecisionCandidateIterationModels and add invalid candidate-iteration cases.

Addressed in dfd10eec and extended with behavior-level coverage in ebcb0c90.

The shared quality-score validator is now used for canonical decision model refs, runtime model refs, and candidateIterations[].models[].

Validation tests cover negative values, values above 1, and non-finite .nan values.

The routing-strategies E2E now verifies both model-card fallback and reversed decision-scoped quality overrides through real chat-completion requests; the existing cost policy remains covered.

make agent-ci-gate ENV=cpu and the targeted isolated Podman/Kind E2E pass.

…cision-candidate-quality Signed-off-by: 0027020836 <du.changyong2@iwhalecloud.com>

Signed-off-by: 0027020836 <du.changyong2@iwhalecloud.com>

feat(router): support decision-scoped candidate quality

b057b55

Signed-off-by: 0027020836 <du.changyong2@iwhalecloud.com>

duchangyong requested review from Xunzhuo and rootfs as code owners July 28, 2026 12:52

github-actions Bot assigned AayushSaini101, drivebyer, FAUST-BENCHOU, ramkrishs, shraderdm and WUKUNTAI-0211 Jul 28, 2026

github-actions Bot deleted a comment from codecov-commenter Jul 28, 2026

Xunzhuo reviewed Aug 1, 2026

View reviewed changes

0027020836 added 2 commits August 1, 2026 14:52

Merge remote-tracking branch 'origin/main' into feature/issue-2711-de…

888b45e

…cision-candidate-quality Signed-off-by: 0027020836 <du.changyong2@iwhalecloud.com>

fix(config): validate candidate iteration quality scores

dfd10ee

Signed-off-by: 0027020836 <du.changyong2@iwhalecloud.com>

github-actions Bot deleted a comment from codecov-commenter Aug 1, 2026

test(e2e): cover decision-scoped quality overrides

ebcb0c9

Signed-off-by: 0027020836 <du.changyong2@iwhalecloud.com>

github-actions Bot deleted a comment from codecov-commenter Aug 1, 2026

duchangyong requested a review from Xunzhuo August 1, 2026 21:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Router][CLI][Dashboard][Docs] Support decision-scoped candidate quality - #2714

[Router][CLI][Dashboard][Docs] Support decision-scoped candidate quality#2714
duchangyong wants to merge 4 commits into
vllm-project:mainfrom
duchangyong:feature/issue-2711-decision-candidate-quality

duchangyong commented Jul 28, 2026 •

edited

Loading

Uh oh!

netlify Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

Xunzhuo Aug 1, 2026

Uh oh!

duchangyong Aug 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

duchangyong commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

netlify Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vllm-semantic-router ready!

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👥 vLLM Semantic Team Notification

📁 config

📁 dashboard

📁 deploy

📁 e2e

📁 src/semantic-router

📁 src/vllm-sr

📁 website

🎉 Thanks for your contributions!

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Supply Chain Security Report — All Clear

Uh oh!

Xunzhuo Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

duchangyong Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

duchangyong commented Jul 28, 2026 •

edited

Loading

netlify Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

📁 `config`

📁 `dashboard`

📁 `deploy`

📁 `e2e`

📁 `src/semantic-router`

📁 `src/vllm-sr`

📁 `website`

github-actions Bot commented Jul 28, 2026 •

edited

Loading