[Epic] Integrate Switchyard routing into NemoClaw-managed Hermes through native NeMo Relay

[cjagwani]

Scope decision — accepted

Canonical NemoClaw scope was accepted in the 2026-08-13 maintainer decision recorded in this issue. The Hermes-first integration may proceed as dependency-aware implementation work. General availability remains gated on stable released upstream artifacts and all compatibility, credential, policy, lifecycle, and E2E criteria below.

Problem

NemoClaw can run Hermes, and the upstream native integration is converging, but NemoClaw does not yet provide an approved, released, secure, and end-to-end validated way to enable Switchyard model routing for a managed Hermes sandbox.

Without that integration, users must choose compatible upstream revisions, build or install experimental plugin code, write Relay and Switchyard configuration, attach multiple provider endpoints and credentials, widen network policy, diagnose silent activation failures, and decide retry/fallback behavior themselves.

V5 goal

A user explicitly enables an approved Switchyard routing configuration for a NemoClaw-managed Hermes agent. Normal Hermes turns then use the released native path:

• no Relay CLI wrapper;
• no switchyard-server sidecar;
• no upstream source checkout or ad hoc build;
• at least two real target models behind OpenShell-secured provider boundaries;
• persistent, fail-closed activation;
• canonical non-secret routing evidence; and
• restart, rebuild, cleanup, and negative-security qualification.

The first milestone qualifies seeded weighted-random and LLM-classifier routing. Stage routing, response-judging escalation, and other routers require separate qualification before being called supported.

Exact architecture and ownership

NemoClaw-managed OpenShell sandbox
└── normal supervised Hermes process
    └── native NeMo Relay runtime
        └── released nvidia.switchyard plugin
            └── OpenShell-secured providers
                ├── judge/classifier, when required
                ├── weak/efficient target
                └── strong/capable target

Relay intercepts and replaces Hermes downstream LLM execution with the Switchyard plugin async generator. Switchyard produces the RoutingOutcome, rewrites the request, and executes the selected target. Relay owns plugin discovery, activation, correlation, and lifecycle. Switchyard owns the routing algorithm and selected-target execution. Everything runs in the supervised Hermes process with zero Relay or Switchyard sidecars.

Within this accepted scope, NemoClaw owns:

1. selecting and verifying a released compatible Hermes, Relay, Switchyard, plugin, and OpenShell set;
2. installing verified artifacts for supported Linux architectures without source builds;
3. generating the released two-file activation/deployment contract and failing readiness when requested routing is inactive;
4. provisioning judge, weak, and strong provider identities through OpenShell without exposing raw credentials;
5. enforcing network, redirect, timeout, retry, fallback, cancellation, and stream-failure policy;
6. exposing bounded non-secret active-state and per-turn routing evidence; and
7. managed E2E, security, upgrade, rollback, operations, and support documentation.

NemoClaw does not own Switchyard algorithms, provider HTTP transport, a fork of the Relay plugin, benchmark claims, or a second harness-specific routing implementation.

Readiness update — 2026-08-25

The upstream pieces are closer, but there is still no stable released compatibility tuple.

Dependency	Current evidence	NemoClaw implication
Hermes loader	NousResearch/hermes-agent#77915 merged and is present in the latest Hermes release	The native in-process loader is released, but the release still selects Relay 0.7.x
NeMo Relay	0.7.3 is stable; 0.8.0-rc.3 is prerelease	Relay 0.8 plugin identity/config changes cannot be a production pin yet
Switchyard native plugin	NVIDIA-NeMo/Switchyard#528 is open, review-required, and targets Relay 0.8.0-rc.3	Moving PR code is not the V5 artifact; its final manifest/config contract can still change
Switchyard release	v0.2.0 is stable but has no native plugin bundle release asset	A version number alone cannot install or verify the native integration
Plugin packaging	#528 contains materialization work but no stable provenance-bound release bundle for both linux/amd64 and linux/arm64	V5 packaging remains blocked
Routing telemetry	Relay structured plugin identity work has landed on the 0.8 release line; NVIDIA/NeMo-Relay#888 remains open for typed OTLP metric-mark/redaction correctness	Canonical switchyard.routing.decision evidence still needs a stable qualified contract
OpenShell	v0.0.113 is stable; NVIDIA/OpenShell#2722 remains open	Fresh provider creation can be evaluated, but safe identity-bound reuse, mutation, and credential rotation cannot yet be claimed

The production dependency-selection and merge gate opens only when #8885 accepts a stable, immutable, provenance-bound tuple. Dependency-aware draft implementation may proceed without turning unreleased inputs into active selectors. PR heads, prereleases, source builds, latest URLs, and unbound artifacts do not satisfy that gate.

V0–V5 evidence ledger

Iteration	Architecture	Result	Meaning
V0	Relay 0.6 sidecar plus separate switchyard-server	Superseded	Historical compatibility path only
V1	Relay CLI wrapper launches one-off Hermes; Switchyard is statically linked	Passed; frozen at d58276a	Proved the components could classify, choose semantic targets, and stream inside a managed sandbox
V2	Supervised Hermes loads Relay and the external Switchyard plugin in-process	Passed; frozen at 3c6932b	Proved the intended zero-sidecar process topology and repeated turns
V3	V2 plus final calls through inference.local	Passed; frozen at 28749b8; repaired at 34d1b06	Proved gateway-owned credential substitution while exposing the single-host-model limitation
V4 demo	Native in-process path plus three attached OpenShell providers, two real cloud targets, route watcher, and read-only UI label	Passed as experimental evidence; frozen to Switchyard #270 commit c69a8b68f7c85e4b610c077690f90db6de9053ed	Proved the desired experience and security shape, not a released support contract
V5	Stable released artifacts, accepted scope, canonical configuration, secure providers, failure policy, E2E, and docs	Not started as canonical product code	Final supported integration if approved

Preserve V4 on its qualified immutable commit. Do not replace it with a moving PR head. Remove prototype overlays only after the released V5 replacements pass equivalent and stronger evidence.

Delivery tracker

Work package	Deliverable	Issue	Current state
Product decision	Canonical NemoClaw vs Community Solutions; lifecycle, compatibility, security, E2E, and support owners	This issue	Accepted 2026-08-13
Released compatibility and packaging	Stable tuple, exact locks, two architectures, immutable hashes, producer provenance, no source builds	#8885	Blocked on stable upstream bundle
Activation and readiness	Released manifest-backed Relay activation plus Switchyard deployment; exact route identity; fail-closed readiness; zero sidecars	#8886	Contract prepared; blocked on released tuple
Secure provider boundary	Separately attached judge/weak/strong providers; gateway-authored evidence; no raw credentials	#8887	Fresh-only recommendation prepared; mutation/rotation blocked on OpenShell #2722
Network and failure behavior	Destination policy, redirect/body/SSE bounds, deadlines, retries, fallback, cancellation, redaction	#8888	Contract prepared; upstream transport gaps remain
Managed qualification	Released topology, two real targets, canonical decision events, restart/rebuild, negative security, exact evidence and teardown	#8889	Contract prepared; waits on prior gates
Documentation and operations	Compatibility matrix, enable/disable, upgrade/rollback, limitations, ownership, troubleshooting	Follow-up after #8889	Not started
Additional routers	StageRouter and response-judging escalation behavior/evaluation	Separate follow-ups	Out of first milestone
Other harnesses	Reusable Relay-host contract and gaps for OpenClaw/Deep Agents	Separate design after Hermes	Out of Hermes-first scope

Configuration and provider decisions

The released path uses two explicit files:

1. a manifest-backed Relay activation file containing plugin identity, native API/ABI compatibility, immutable entrypoint/bundle identity, and deployment path; and
2. a Switchyard v1 deployment document containing llm_clients, targets, and routes.

NemoClaw must reject the older plugin_id-only form, unknown route IDs, incompatible API/ABI/ranges, missing assets, and requested-but-inactive routing before any provider request.

For the first V5 provider design, use three separately attached OpenShell providers: judge, weak, and strong. Keep inference.local as Hermes's ordinary bootstrap/default route; do not use it for Switchyard targets because it represents one gateway-wide host-selected model.

Each Switchyard llm_client receives an endpoint-scoped raw-key placeholder through api_key_env; Switchyard constructs the provider Authorization header. forward_auth remains false. Caller authorization, cross-provider placeholder use, direct-provider egress, and silent fallback to inference.local must be denied.

Fresh creation of uniquely named providers is the initial boundary. Existing-provider reuse, update, mutation, and credential rotation remain blocked until OpenShell provides identity-bound conditional mutation and secret-free endpoint/profile attestation or an accepted equivalent.

Supported acceptance criteria

• Maintainers accepted canonical product scope and the documented ownership boundary on 2026-08-13.
• A fresh sandbox consumes only stable, compatible, immutable, provenance-verified artifacts for every supported architecture.
• Normal supervised Hermes loads Relay and nvidia.switchyard persistently with zero routing sidecars.
• Requested routing becomes active and ready before the first provider request.
• Repeated Hermes turns select at least two distinct real target models.
• Provider credentials remain outside agent-visible files, environment, argv, logs, errors, telemetry, and artifacts and resolve only at intended endpoints.
• Unauthorized egress, caller-auth forwarding, cross-endpoint use, and routing bypass fail closed.
• Invalid activation, missing/corrupt plugin, provider denial, redirect, timeout, retry exhaustion, fallback, cancellation, and post-first-stream failure are qualified.
• Buffered, streaming, tool, session, restart, rebuild, cleanup, and redaction behavior remain correct.
• Every routing assertion comes from the canonical switchyard.routing.decision event correlated to the Hermes turn—not reconstructed provider logs.
• Machine-readable evidence binds expected cases, observed pass status, exact artifacts/digests, architecture, topology, provider identities, route correlation, redaction scan, and teardown.
• Documentation names supported versions, routers, ownership, enable/disable, limitations, upgrade, rollback, and troubleshooting.

Constraints and non-goals

• Do not rewrite or vendor Switchyard routing algorithms in NemoClaw.
• Do not harden the Relay CLI/static-plugin or Decision API sidecar prototypes into the supported default.
• Do not treat merged source, a prerelease, a green partial CI run, or a working demo as a released compatibility set.
• Do not put raw provider credentials in the sandbox.
• Do not broaden the first milestone to every Relay feature, Switchyard router, model recipe, or agent harness.
• [Epic] Deliver the Hermes Harness Optimization Blueprint with NeMo Platform Insights #8501 owns benchmark-backed Hermes recipes; this Epic owns the routing/security/platform contract.

Other agent harnesses

The platform design can extend beyond Hermes only when another harness provides equivalent Relay lifecycle, downstream model-call interception, configuration, routing metadata/feedback, and credential-bound provider execution. NemoClaw should reuse the secure provider and policy layers where possible, but each harness needs separately accepted adapter/lifecycle scope and E2E evidence. Hermes is first because the native loader exists upstream.

Remaining decisions

• Who owns compatibility, security review, live E2E, release qualification, and support lifecycle?
• Which future stable Hermes/Relay/Switchyard/plugin/OpenShell tuple is the first supported record?
• What released producer publishes the native bundle for both required architectures with immutable provenance?
• Is fresh-only provider provisioning acceptable for the first milestone, or must OpenShell docs: add Brev troubleshooting guide #2722 land first?
• Which seeded random/classifier recipe is the first supported preset, and who owns its benchmark evidence?
• What exact upstream fallback behavior is accepted before and after the first streamed event?

Discovery and coordination

• Will Curran: confirmed the Hermes-first goal and requested assessment for other harnesses.
• Sean Lopp: provided Hermes integration and setup/runtime UX context and directed contract questions to Michael Demoret.
• Michael Demoret: identified Relay as the intended Hermes integration surface.
• Bryan Bednarski: confirmed the in-process direction and release sequencing.
• Ryan Lempka: provided Switchyard configuration, benchmarking, metadata, and library-mode context.
• Apurv Kumaria: Security Observer's Deep Agents weak/strong prototype informed demo clarity and secure provider handling.
• Aaron Erickson: requested this Epic and the concrete issues/PR plan.
• Supporting material from Alex Fournier, the NeMo Relay team, the Switchyard team, and the Hermes team was also reviewed.

Related work

• [Epic] Deliver the Hermes Harness Optimization Blueprint with NeMo Platform Insights #8501 owns benchmark-backed Hermes optimization recipes that may consume this platform capability.
• Draft feat(hermes): define native Switchyard routing contract #9012 is useful lifecycle/prototype evidence but is not the final released configuration or packaging contract.

Checklist

• Prototype evidence and limitations are preserved.
• Current upstream release state and exact remaining gates are documented.
• NemoClaw, Relay, Switchyard, Hermes, OpenShell, and benchmark ownership are separated.
• Compatibility, activation, provider, failure-policy, and E2E child issues are concrete.
• The frozen V4 demo is separated from the released V5 path.
• Canonical NemoClaw scope accepted on 2026-08-13.

[cjagwani]

Prototype evidence — 2026-07-30

Status: upstream and Hermes-image plumbing passed; OpenShell-managed sandbox execution remains pending. This is disposable source-build evidence, not a supported compatibility claim.

Immutable identities

• NeMo Relay PR fix(onboard): avoid piped sandbox connect sync in step 6 #586 head: 44438179c37cfb214901fd80c126605e075f3935 (refs/pull/586/head)
• Switchyard crates consumed by that Relay lockfile: aa0a31126cc7ca922a6acdf4f1ef656744698c6c
• Pinned Linux builder: rust:1.96.1-trixie@sha256:1f0dbad1df66647807e6952d1db85d0b2bda7606cb2139d82517e4f009967376
• NemoClaw Hermes base: ghcr.io/nvidia/nemoclaw/hermes-sandbox-base@sha256:c4899e00817acb3421037efba013c720a79895bc8c4448b47aa9aca0def3104b (Hermes 0.19.0)
• Built Relay CLI SHA-256: 322ff946408a2cf8d1260ce97af72e960a507225a735046dcecd500c0efc9d65
• Build command: cargo build --locked --release -p nemo-relay-cli --features switchyard

Do not substitute Switchyard PR #192 current head for the Cargo-locked revision. The Relay branch currently wires random and llm_classifier; it does not wire StageRouter.

Passed checks

1. Weighted random, buffered + streaming: nine loopback-only requests exercised both provider/fast and provider/quality; the streaming response and provider system_fingerprint survived Relay/Switchyard translation.
2. LLM classifier, buffered + streaming: the exact provider sequence was provider/classifier → provider/fast for each request, including structured classifier input.
3. Hermes wrapper: Hermes 0.19.0 ran one streaming turn through nemo-relay run --agent hermes; Switchyard selected provider/quality.
4. Routing evidence: ATOF contained switchyard.routing.requested, switchyard.routing.call, and switchyard.routing.decision.
5. Isolation: the container had external networking disabled; only a loopback fake provider was available. No Switchyard server process ran.
6. Secret/lifecycle checks: a dummy credential marker did not appear in provider capture, ATOF, or the combined Hermes/Relay log. The Relay process and process-private Hermes overlay were gone after wrapper exit.

What this proves

The source-built CLI path works as:

NemoClaw Hermes 0.19 base
└── Relay CLI transparent wrapper
    ├── Hermes child process
    └── in-process Switchyard libsy random/classifier routing
        └── Relay-owned loopback provider dispatch

It supports the proposed reusable boundary: Relay/Switchyard owns routing and provider dispatch, while Hermes supplies a harness adapter. Another harness can reuse the same core only after it supplies equivalent invocation, metadata/feedback, lifecycle, configuration, and credential hooks.

Remaining gates and gaps

• This was the pinned NemoClaw Hermes base image, not an OpenShell-managed sandbox. The available test environment had no registered OpenShell gateway, so upload/exec, managed policy, and sandbox lifecycle remain unproven.
• The stock final Hermes image cannot build Relay in place; the prototype should upload this prebuilt, architecture-matched artifact into a disposable workspace. Canonical support must consume a published package instead.
• Hermes startup produced nine nonfatal transparent_proxy_authentication rejections before the authenticated routed call succeeded. Confirm whether these are expected unauthenticated model-discovery probes and whether upstream should suppress or authenticate them.
• Real multi-provider endpoints, Providers v2 credentials, endpoint binding, fallback failure injection, restart/recovery, and policy enforcement remain untested.
• StageRouter remains outside the current in-process branch adapter.

Next prototype step

Run the same pinned binary, config, and fake provider inside a disposable NemoClaw-managed Hermes sandbox without widening network policy; then repeat failure/fallback and cleanup assertions. After that, use the findings only to define the native released-path work.

[cjagwani]

Architecture checkpoint — V1 complete, V2 separated

The managed behavioral prototype is now complete. It remains pinned to Relay PR #586 (44438179c37cfb214901fd80c126605e075f3935) and Switchyard (aa0a31126cc7ca922a6acdf4f1ef656744698c6c) as an immutable record of the architecture available when the experiment began.

Final managed evidence:

• Hermes 0.19.0 ran inside a disposable NemoClaw/OpenShell-managed sandbox.
• A bounded status prompt selected the weak/fast target.
• A fail-closed remediation prompt selected the strong/quality target.
• Classifier and routed provider responses streamed through Relay back to Hermes.
• Switchyard request, call, and decision marks were captured.
• All model providers were deterministic loopback fixtures; no real credential was used.
• Focused tests passed (14/14), and the sandbox, gateway, processes, and temporary artifacts were cleaned up.

This proves the routing behavior and managed-sandbox feasibility. It does not prove the intended always-on architecture or a released compatibility contract.

Upstream has since converged on a different production topology:

• Hermes #77915 loads Relay and dynamic plugins in the normal Hermes process.
• Switchyard feat: recommend optimal NIM model based on detected GPU VRAM #270 supplies the external nvidia.switchyard plugin using Relay native API v1 and Switchyard-owned provider HTTP dispatch.
• Relay fix(onboard): avoid piped sandbox connect sync in step 6 #586 and Switchyard fix: un-exclude nemoclaw/dist in .dockerignore for sandbox builds #220 are superseded behavioral references. Relay docs: add sandbox policy configurability section to README #594 is not required by the feat: recommend optimal NIM model based on detected GPU VRAM #270 design.

V1 will not be rewritten to resemble V2. The next prototype will use a separate branch/worktree and will prove the normal supervised Hermes process, persistent Relay/plugin lifetime, deterministic weak/strong routing, restart, and cleanup. A later lane will test real models through inference.local without placing provider credentials in the sandbox.