Add routing_method experiment variable for A/B/C hydra experiment#308273
Merged
Add routing_method experiment variable for A/B/C hydra experiment#308273
Conversation
d02b243 to
1c93551
Compare
Contributor
Screenshot ChangesBase: Changed (8) |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an experiment-driven “routing method” selector for Auto Mode’s model router integration, enabling the auto-intent-service to vary its routing algorithm per request (e.g., binary vs hydra), and handling server-signaled router fallbacks.
Changes:
- Introduces
chat.advanced.autoModeRoutingMethodas a team-internal, experiment-based setting (ECS:copilotchat.autoModeRoutingMethod). - Sends
routing_methodin the router request body and extends the router response type to include fallback/hydra-related fields. - Handles
fallback=truerouter responses in Auto Mode by falling back to default model selection and logging the fallback reason.
Show a summary per file
| File | Description |
|---|---|
| extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts | Extends router response typing and adds routing_method to the request + extra trace logging. |
| extensions/copilot/src/platform/endpoint/node/automodeService.ts | Reads the experiment variable, forwards routingMethod to the router, and handles router-signaled fallback behavior. |
| extensions/copilot/src/platform/configuration/common/configurationService.ts | Registers the new experiment-based config key for selecting the routing method. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 4
extensions/copilot/src/platform/configuration/common/configurationService.ts
Outdated
Show resolved
Hide resolved
extensions/copilot/src/platform/endpoint/node/automodeService.ts
Outdated
Show resolved
Hide resolved
bhavyaus
commented
Apr 7, 2026
justschen
approved these changes
Apr 7, 2026
Port of microsoft/vscode-copilot-chat#4947. - Add AutoModeRoutingMethod experiment-based setting - Extend RouterDecisionResponse with HYDRA/fallback metadata - Pass routing_method in router request body when set by ECS - Handle fallback=true responses in AutomodeService
- Add routingMethod, fallback, fallbackReason to automode.routerDecision telemetry - Update GDPR annotation to include 'fallback' in predictedLabel values - Fix AutoModeRoutingMethod JSDoc to clarify it only controls routing_method value, not whether the router is called (gated by UseAutoModeRouting) - Fix fallbackReason high-cardinality: use bounded 'routerFallback' value instead of embedding server-provided fallback_reason string
1f5630a to
008b8b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of microsoft/vscode-copilot-chat#4947.
Summary
Add
copilotchat.autoModeRoutingMethodECS flight variable to control which routing algorithm the auto-intent-service uses per-request. This enables A/B/C experiments for the HYDRA model router.A/B/C Experiment Design
Fallback for B/C → automod (path A), signaled via
fallback=truein the response.Changes
AutoModeRoutingMethodexperiment-based settingroutingMethodparam, sendrouting_methodin request body, extendRouterDecisionResponsewith new fieldsfallback=trueresponses