Skip to content

Commit 2ab971c

Browse files
committed
feat(relay): add isolated Switchyard routing plugin
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
1 parent 3948b4c commit 2ab971c

16 files changed

Lines changed: 7884 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,20 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
66

77
## [Unreleased]
88

9+
### Added
10+
11+
- **NeMo Relay native plugin** — a dynamically loaded integration that runs
12+
libsy's weighted-random, LLM-classifier, escalation, and stage-router
13+
algorithms in process while Switchyard owns provider HTTP dispatch,
14+
credentials, translation, retries, and fallback. Managed calls require NeMo
15+
Relay 0.7 or newer and do not depend on `switchyard-server`.
16+
17+
- **NeMo Relay routing-model usage marks** — classifier judges, escalation
18+
judges and discarded weak candidates, and failed routing candidates now emit
19+
`switchyard.routing.llm_call` ATOF marks with normalized token usage and
20+
latency. The final serving call remains represented only by Relay's outer LLM
21+
lifecycle event to prevent double-counting.
22+
923
### Removed
1024

1125
- **Latency-aware router** — the `latency_service` route type and its
@@ -31,6 +45,14 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
3145

3246
### Fixed
3347

48+
- **NeMo Relay stage and escalation integration** — preserved same-protocol
49+
request bodies are now re-encoded after tier prompts or handoff notes mutate
50+
the normalized request; Relay's synthetic `gateway-gateway` identity no
51+
longer shares escalation latch state across unrelated raw gateway requests;
52+
stage decision marks now retain picker-default tiers, decision sources, and
53+
hard-override confidence. Target bindings also accept non-secret
54+
`extra_body` defaults for provider-specific judge controls.
55+
3456
- **Response `model` now names the model that actually served the request**, on
3557
every serving path and wire format. Streamed Anthropic and Responses replies,
3658
and every libsy-served reply, previously echoed the model id the client

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ algorithm you write yourself.
2121
- **Protocol Translation**: convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
2222
- **Multi-Backend Routing**: random routing, LLM-as-classifier routing, signal-driven stage-router, or your own algorithm
2323
- **Operational Metrics**: Prometheus metrics cover requests, errors, latency, tokens, and routing overhead
24+
- **NeMo Relay Plugin**: run random, classifier, escalation, or stage routing in Relay while Switchyard owns provider HTTP dispatch
2425

2526
## Maturity
2627

@@ -154,6 +155,7 @@ configured LLM client selects one upstream format.
154155
- **[`switchyard-libsy`](crates/libsy/README.md)**: embed routing algorithms in a Rust application
155156
- **[`switchyard-protocol`](crates/protocol/README.md)**: provider-neutral request, response, and streaming types
156157
- **[`switchyard-translation`](crates/switchyard-translation/README.md)**: request, response, and stream translation
158+
- **[`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md)**: install Switchyard as a native NeMo Relay plugin
157159

158160
## Community
159161

0 commit comments

Comments
 (0)