Skip to content

feat(benchmark): add reproducible NeMo Gym routing comparison - #595

Draft
afourniernv wants to merge 1 commit into
mainfrom
codex/sygh-96-nemo-gym-comparison
Draft

feat(benchmark): add reproducible NeMo Gym routing comparison#595
afourniernv wants to merge 1 commit into
mainfrom
codex/sygh-96-nemo-gym-comparison

Conversation

@afourniernv

Copy link
Copy Markdown
Contributor

What

Adds benchmark/nemo_gym/, a version-pinned workflow for comparing two Switchyard routing conditions on the same NeMo Gym MMLU-Redux tasks:

  • strong-only always uses the strong target.
  • policy-model uses an LLM classifier to select the efficient or strong target.

The runner builds the current checkout's switchyard-server, gives each condition a fresh proxy and output directory, and captures Gym rollouts alongside Switchyard statistics and metrics. The comparator pairs matching task/repeat rows before reporting reward, answer and classifier tokens, latency, model totals, and classifier fail-opens.

This adds no production dependency, API, or request-path change.

Why

Gym already provides the switchyard_model adapter, but Switchyard did not have an executable repository example for using it to compare a routed strategy with a fixed-model baseline. This makes that workflow reproducible from a Switchyard checkout and keeps failed or mismatched rollouts visible in the comparison.

Closes #559

Linear: SYGH-96

Builds on NVIDIA-NeMo/Gym#2141, which added Gym's Switchyard model-server adapter.

Notes for reviewers

Start with benchmark/nemo_gym/README.md for the workflow, then run.sh for process and artifact ownership, and compare.py for pairing and accounting semantics.

The example uses attached mode so it exercises the current Switchyard checkout rather than Gym's pinned released wheel. It starts a fresh proxy per condition because router state and /v1/stats are process-wide, and it snapshots statistics before shutdown because Gym's attached-mode snapshot is best-effort.

Validation:

  • env -u OPENROUTER_API_KEY -u NVIDIA_API_KEY -u OPENAI_API_KEY -u ANTHROPIC_API_KEY uv run pytest tests/ -q -m "not integration" — 121 passed, 2 deselected, 2 subtests passed
  • uv run ruff check .
  • uv run mypy switchyard
  • bash -n benchmark/nemo_gym/run.sh
  • git diff --check
  • cargo run --release -p switchyard-server -- --config benchmark/nemo_gym/routes.toml --dry-run
  • Live NVIDIA smoke with one matching MMLU-Redux task per condition; both routes completed and classifier and answer usage were captured separately

The limited live run validates the integration and accounting only; it is not presented as a model-quality benchmark.

Signed-off-by: Alex Fournier <afournier@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a reproducible NeMo Gym example for comparing routing strategies

1 participant