[feature] Add VGSR as a first-class routing algorithm

[gburachas]

Problem

Switchyard already offers several useful routing strategies, but there is an opportunity to add a purpose-built algorithm for the local-first quality/cost frontier.

The proposed VGSR algorithm is intended to make a more informed escalation decision than fixed routing or a simple tier split: retain requests that a local target can handle well, and use a stronger cloud target only when its added capability is likely to improve the outcome materially.

If that decision policy performs as intended, it could improve the practical quality-to-cloud-cost trade-off for hybrid deployments: higher answer quality than an all-local baseline, with substantially less cloud use than an all-cloud baseline.

This is a concrete algorithm-extension request, not a request for another general routing framework. The aim is to evaluate and expose VGSR as a first-class Switchyard option, with clear benchmarks against existing strategies such as passthrough, random routing, and LLM-classifier routing.

Proposed solution

Add VGSR as a first-class Switchyard routing algorithm.

VGSR would provide a local-first route that selects among configured local and cloud targets. The user configures their available targets and one public route; Switchyard applies VGSR to decide whether the request should remain local or be sent to a cloud-capable target.

The user-facing configuration should be a native deployment-TOML route type:

[routes.local_first]
id = "switchyard/local-first"
type = "vgsr"

local_targets = ["local_fast", "local_capable"]
cloud_targets = ["cloud_capable"]

The exact VGSR options should be documented with clear defaults and validation rules. Users should not need to understand the algorithm’s internal implementation to configure or operate the route.

The implementation should be native Rust and use the same Switchyard execution model as other routing algorithms:

• switchyard-libsy exposes a public VgsrRouter for embedded Rust applications.
• switchyard-runner accepts type = "vgsr" and constructs the router from deployment TOML.
• switchyard-server makes the configured route available to OpenAI- and Anthropic-compatible clients.
• Existing Python/PyO3 users reach the algorithm through the normal native route-construction path.

VGSR should select targets and report its routing decision. Switchyard remains responsible for provider communication, protocol translation, retries, streaming, and configured fallback handling.

Alternatives considered

Use an existing LLM classifier route

An LLM classifier can select between local and cloud targets, and remains appropriate for general-purpose model selection.

However, it requires a separate model call to make the routing decision and exposes the user to classifier-prompt design and classifier-model behavior. VGSR would offer a dedicated, documented local-first routing policy as a standard Switchyard algorithm.

Run VGSR as a separate proxy or service

A separate service could return a model-selection decision before Switchyard processes the request.

This adds another deployment, network hop, operational dependency, and integration boundary. It would also make VGSR unavailable to users embedding Switchyard directly as a Rust library. A native algorithm keeps the routing policy within the existing Switchyard runtime.

Route all requests locally or all requests to the cloud

Both remain valid simple configurations, but neither adapts to request difficulty. VGSR is intended for users who want a controlled balance between local-model usage and cloud-model capability.

Scope notes

• The primary change belongs in the routing-algorithm layer: a new native VgsrRouter implementing the existing libsy algorithm interface.
• The deployment configuration surface changes: native TOML gains type = "vgsr" and VGSR-specific configuration fields.
• The public Rust API changes: switchyard-libsy exposes the router and its typed configuration.
• Server API routes and provider translation codecs should not need VGSR-specific behavior. The server should expose VGSR routes through its existing model-discovery and request-serving paths.
• The implementation should use existing Switchyard model-call, fallback, telemetry, and translation mechanisms rather than adding a VGSR-specific HTTP client.
• Existing route types and deployment TOML must continue to work unchanged. VGSR is additive.
• Security and sandboxing are not part of this proposal.

Additional context

This proposal is related to:

• [Feature] Support classifier routing across arbitrary named, purpose-driven targets #240, “Support classifier routing across arbitrary named, purpose-driven targets,” because VGSR also needs to select among configured named targets.
• [feature] Extensible stage router known tools #333, “Extensible stage router known tools,” because both proposals aim to make routing decisions more useful across different application workloads.

Suggested acceptance criteria:

• A valid type = "vgsr" route passes switchyard-server --dry-run.
• Rust applications can construct and execute VgsrRouter through switchyard-libsy.
• A configured VGSR route appears in /v1/models.
• OpenAI- and Anthropic-compatible clients can use the same VGSR route.
• Tests cover routine local selections, cloud escalations, unavailable selected targets, and configured fallbacks.
• Logs, traces, and metrics identify the selected target, whether cloud escalation occurred, fallback use, and routing overhead without recording request content.
• Documentation includes a complete local-first example and guidance for evaluating answer quality, cloud use, latency, and routing overhead.

[grahamking]

@gburachas Please expand "VGSR" on first use. What does it stand for?

[grahamking]

Adding a note that this is being discussed internally with @ayushag-nv and @ryan-lempka . Issue is active.