Skip to content

Add routing_method experiment variable for A/B/C hydra experiment#308273

Merged
bhavyaus merged 2 commits intomainfrom
dev/bhavyau/hydra-routing-method
Apr 7, 2026
Merged

Add routing_method experiment variable for A/B/C hydra experiment#308273
bhavyaus merged 2 commits intomainfrom
dev/bhavyau/hydra-routing-method

Conversation

@bhavyaus
Copy link
Copy Markdown
Collaborator

@bhavyaus bhavyaus commented Apr 7, 2026

Port of microsoft/vscode-copilot-chat#4947.

Summary

Add copilotchat.autoModeRoutingMethod ECS flight variable to control which routing algorithm the auto-intent-service uses per-request. This enables A/B/C experiments for the HYDRA model router.

A/B/C Experiment Design

Path Value Behavior
A '' (empty, default) Production automod — no router call
B 'binary' Binary classifier v1 (current router)
C 'hydra' HYDRA multi-head capability matching (new)

Fallback for B/C → automod (path A), signaled via fallback=true in the response.

Changes

  • configurationService.ts: New AutoModeRoutingMethod experiment-based setting
  • routerDecisionFetcher.ts: Add routingMethod param, send routing_method in request body, extend RouterDecisionResponse with new fields
  • automodeService.ts: Read experiment var, handle fallback=true responses

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2026

Screenshot Changes

Base: fc99d59e Current: 8b35d402

Changed (8)

chat/chatQuestionCarousel/SingleTextQuestion/Light
Before After
before after
chat/chatQuestionCarousel/MultipleQuestions/Light
Before After
before after
chat/aiCustomizations/aiCustomizationManagementEditor/McpServersTab/Dark
Before After
before after
chat/aiCustomizations/aiCustomizationManagementEditor/InstructionsTab/Light
Before After
before after
chat/aiCustomizations/aiCustomizationManagementEditor/PluginsTab/Dark
Before After
before after
chat/aiCustomizations/aiCustomizationManagementEditor/PromptsTabScrolled/Dark
Before After
before after
chat/aiCustomizations/aiCustomizationManagementEditor/PromptsTabScrolled/Light
Before After
before after
chat/aiCustomizations/aiCustomizationManagementEditor/McpServersTabNarrow/Light
Before After
before after

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an experiment-driven “routing method” selector for Auto Mode’s model router integration, enabling the auto-intent-service to vary its routing algorithm per request (e.g., binary vs hydra), and handling server-signaled router fallbacks.

Changes:

  • Introduces chat.advanced.autoModeRoutingMethod as a team-internal, experiment-based setting (ECS: copilotchat.autoModeRoutingMethod).
  • Sends routing_method in the router request body and extends the router response type to include fallback/hydra-related fields.
  • Handles fallback=true router responses in Auto Mode by falling back to default model selection and logging the fallback reason.
Show a summary per file
File Description
extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts Extends router response typing and adds routing_method to the request + extra trace logging.
extensions/copilot/src/platform/endpoint/node/automodeService.ts Reads the experiment variable, forwards routingMethod to the router, and handles router-signaled fallback behavior.
extensions/copilot/src/platform/configuration/common/configurationService.ts Registers the new experiment-based config key for selecting the routing method.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 4

bhavyaus added 2 commits April 7, 2026 11:37
Port of microsoft/vscode-copilot-chat#4947.

- Add AutoModeRoutingMethod experiment-based setting
- Extend RouterDecisionResponse with HYDRA/fallback metadata
- Pass routing_method in router request body when set by ECS
- Handle fallback=true responses in AutomodeService
- Add routingMethod, fallback, fallbackReason to automode.routerDecision telemetry
- Update GDPR annotation to include 'fallback' in predictedLabel values
- Fix AutoModeRoutingMethod JSDoc to clarify it only controls routing_method
  value, not whether the router is called (gated by UseAutoModeRouting)
- Fix fallbackReason high-cardinality: use bounded 'routerFallback' value
  instead of embedding server-provided fallback_reason string
@bhavyaus bhavyaus force-pushed the dev/bhavyau/hydra-routing-method branch from 1f5630a to 008b8b6 Compare April 7, 2026 18:37
@bhavyaus bhavyaus enabled auto-merge (squash) April 7, 2026 21:08
@bhavyaus bhavyaus merged commit 264da2a into main Apr 7, 2026
37 of 38 checks passed
@bhavyaus bhavyaus deleted the dev/bhavyau/hydra-routing-method branch April 7, 2026 21:14
@vs-code-engineering vs-code-engineering bot added this to the 1.116.0 milestone Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants