Skip to content

[Feature] Add typed success estimates and immutable evidence snapshots - #3518

Open
sujal111 wants to merge 1 commit into
vllm-project:mainfrom
sujal111:feat/typed-success-estimates-3480
Open

[Feature] Add typed success estimates and immutable evidence snapshots#3518
sujal111 wants to merge 1 commit into
vllm-project:mainfrom
sujal111:feat/typed-success-estimates-3480

Conversation

@sujal111

@sujal111 sujal111 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Add typed success estimates and an immutable in-memory evidence snapshot under Router Learning (#3480).
Observe mode records the estimates in Router Replay. The selected model does not change.
routing_sampling stays the default — this PR does not add success_constrained or apply mode.

Closes #3480
Related #3412
Related #3494

Purpose

First implementation PR after the proposal. pkg/extproc now freezes the in-memory experience at request start and reports a typed estimate per candidate (calibrated / insufficient_evidence / unsupported / stale / conflict). Seed-only QualitySeed and classifier/similarity scores are not treated as P(success).

wg/mom-routing

Next PRs

Test Plan

  • go test ./pkg/extproc/ -run 'SuccessEstimate|TestRouterLearning|TestBuildReplayRoutingRecordCapturesLearning'
  • Covers seed-only, missing calibration, stale evidence, sparse backoff, concurrency, and observe mode keeping the base model

Test Result

Targeted extproc run passed locally. Full make test-semantic-router is on CI — I could not rebuild the candle bindings here.

Freeze in-memory Router Learning experience at request start and report
conservative typed estimates so later observe-only selection can rank on
calibrated P(success) without treating QualitySeed as a probability.

Signed-off-by: SUJAL MAITI <sujal@SUJALs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
@netlify

netlify Bot commented Sep 5, 2026

Copy link
Copy Markdown

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit a882213
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/6a9c547c16ab460008287682
😎 Deploy Preview https://deploy-preview-3518--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions github-actions Bot added pr/needs-review Ready for reviewer attention. wg/mom-routing Owned by the MoM and Routing Workgroup. pr/needs-rebase Needs rebase or conflict resolution. and removed pr/needs-review Ready for reviewer attention. labels Sep 5, 2026

@Xunzhuo Xunzhuo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One core part of #3480 is not wired into the observe path yet. attachSuccessEstimateObserveDiagnostics calls estimateCandidateSuccess with no stale horizon, so StaleAfter stays zero and stale evidence can never produce the stale result. The same path uses resolveExperience, which simply picks the first populated scope; mergeScopedExperience is never called, so conflicting scopes are silently accepted and the conflict result is unreachable. The direct helper tests do not exercise these production paths. Please wire the decision/global observe controls through the real lookup path and add observe-path coverage showing stale and conflicting evidence fail conservatively.

@github-actions github-actions Bot added pr/needs-author Waiting for author changes or response. and removed pr/needs-rebase Needs rebase or conflict resolution. labels Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr/needs-author Waiting for author changes or response. wg/mom-routing Owned by the MoM and Routing Workgroup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Define typed success estimates and immutable evidence snapshots

2 participants