Skip to content

README.md

Latest commit

 

History

History
27 lines (23 loc) · 1.34 KB

README.md

File metadata and controls

27 lines (23 loc) · 1.34 KB
Raw

Dynamo KV-Router A/B Experiment

A rigorous A/B benchmark of NVIDIA Dynamo's KV-aware router vs round-robin routing, on identical multi-worker aggregated deployments, driven by a real production trace (Mooncake) and profiled with a custom observability stack (inferscope) on top of NVIDIA's own AIPerf methodology.

What this is

  • Method: two deployments differing ONLY in --router-mode (round-robin vs kv), same worker count, deterministic KV hashing, warm-up + 3 averaged runs.
  • Workload: Mooncake arxiv trace (real shared-prefix structure; the case where KV routing matters). Filtered to inputs <= 16384 tokens (89.7% of the trace).
  • Measurement: AIPerf for client-side latency/throughput; inferscope --sample-only for server-side per-device GPU truth (SM/VRAM/power) that the NVIDIA A/B guide does not show.
  • Triangulation: AIConfigurator predicted vs AIPerf measured vs inferscope resource-observed.

Status

Cold preparation complete (see RUNBOOK.md). Execution pending on a rented multi-GPU instance. This repo holds the runbook, the A/B launch scripts, and (once run) the analysis.

Layout

  • RUNBOOK.md — the full end-to-end procedure, phase by phase.
  • scripts/arm_common.sh — shared launcher; router mode is the single A/B variable.
  • scripts/arm_off.sh, scripts/arm_on.sh — the two arms (round-robin vs kv).