feat: benchmark for batch-gateway

[lioraron]

Objective

Demonstrate that batch-gateway with gated dispatch protects interactive workload latency while enabling batch workloads to make steady progress toward SLO targets — on shared GPU infrastructure.

The benchmark compares dispatch strategies under a realistic traffic pattern: interactive requests arrive in waves (bursts that saturate the inference server alternating with near-idle periods), while a high-volume batch workload is always available. The key finding should be that gated dispatch fills spare capacity with batch work during idle periods and backs off during bursts, without degrading interactive quality.

Terminology: "Batch" in this spec refers specifically to the OpenAI Batch API interface — clients upload JSONL files via /v1/files, create batch jobs via /v1/batches, and poll for completion. This is distinct from MLPerf's "offline scenario" (all samples available upfront, maximize throughput) and from training-style batch processing. However, scenario 2 (ungated batch with maximum concurrency) is functionally similar to the MLPerf offline scenario and can serve as an upper-bound throughput reference.

Scenarios

#	Scenario	Dispatch mode	Description
0	Interactive only (baseline)	N/A	Only interactive traffic (guidellm burst/idle pattern), no batch workload at all. Establishes the ideal interactive latency baseline — all subsequent scenarios are measured against this.
1	No batch-gateway	N/A	Batch prompts sent as regular interactive requests via a second guidellm instance directly to the inference gateway. Represents what a user would do without batch-gateway.
2	Ungated batch	sync, high concurrency, AIMD disabled	Batch-gateway sends batch requests with aggressive concurrency (`global: 200`, `perEndpoint: 100`, `aimd.enabled: false`). No flow control — batch competes directly with interactive traffic.
3	Gated batch — admission control + AIMD	sync, AIMD enabled, llm-d Router admission control	Batch-gateway sends batch requests with AIMD and `inferenceObjective: "batch-sheddable"` header. The llm-d Router's admission controller rejects sheddable requests (priority < 0) when backend saturation ≥ 1.0, generating 429s that trigger AIMD multiplicative decrease. Active backpressure (admission-based rejection) plus adaptive concurrency (AIMD).
4	Gated batch — flow control + AIMD	sync, AIMD enabled, llm-d Router flow control enabled	Batch-gateway sends batch requests with AIMD and `inferenceObjective: "batch-sheddable"` header. llm-d Router flow control's dispatch cycle dispatches one request per tick, highest priority first — interactive (priority 100) always dispatches before batch (priority -1). This priority ordering passively protects interactive latency. AIMD is enabled on the processor side.
5	Gated batch — async processor	async, prometheus-budget gate	Batch-gateway delegates dispatch to the async-processor, which uses a PromQL-based dispatch budget gate to proactively meter batch requests based on downstream utilization.
6	Gated batch — low concurrency	sync, low `perEndpoint`, AIMD disabled	Batch-gateway sends batch requests with fixed low concurrency (`perEndpoint: 20`, `aimd.enabled: false`). No router-side protection — the only gating mechanism is the processor's concurrency cap. Baseline for measuring the value of smarter protection mechanisms (S3 admission control, S4 flow control).

Note: Scenario 5 depends on the batch-gateway ↔ async-processor integration, which is not yet merged. Scenarios 0–4 and 6 can proceed independently. Scenario 5 will be added once the integration lands.

How to read the results

• Scenario 0 establishes the interactive latency baseline with no batch interference.
• Scenarios 1 and 2 show how much batch workload degrades interactive metrics when there is no gating.
• Scenarios 3–6 show how gated dispatch protects interactive metrics close to the scenario 0 baseline while still making batch progress.

Scenarios 3, 4, and 6: comparing protection mechanisms

• 6 (Low concurrency) is the simplest protection: a fixed perEndpoint cap limits batch parallelism. No adaptive behavior — if the cap is set too high, batch degrades interactive; too low, batch throughput suffers. Baseline for measuring whether smarter mechanisms add real value.
• 3 (Admission control + AIMD) uses the llm-d Router's admission controller to reject batch requests (priority < 0) when saturation reaches 1.0, producing 429s that trigger AIMD multiplicative decrease. AIMD then gradually recovers concurrency during idle periods. Active, adaptive backpressure without flow control's priority dispatch ordering.
• 4 (Flow control + AIMD) uses flow control's dispatch cycle, which dispatches one request per tick, highest priority first. Interactive (priority 100) always dispatches before batch (priority -1). This priority ordering passively protects interactive latency without needing saturation-based rejection. AIMD is enabled but stays flat without a 429 source.

Comparing 6, 3, and 4 shows the value chain: fixed cap → adaptive backpressure → priority-aware scheduling.

For configuration details, see the Flow Control Setup Guide.

Traffic Pattern

Interactive workload

Generated by guidellm in alternating burst/idle phases:

Phase	Rate	Duration	Purpose
Burst	Saturating (e.g. 15 req/s)	60–120s	Simulates peak interactive demand — inference server near capacity
Idle	Low (e.g. 1 req/s)	60–120s	Simulates off-peak — spare capacity available for batch

Multiple cycles (e.g. 3–5 burst→idle) to show the pattern repeating reliably.

Rate sweep (follow-on): Existing llm-d benchmarks sweep request rates (e.g., 1–30 rps) to find the saturation point for each configuration. The initial benchmark uses a fixed burst/idle pattern, but a rate-sweep mode should be added in a later PR to characterize throughput-latency curves under gated vs ungated dispatch.

Warmup: The first burst/idle cycle should be excluded from results to avoid cold-start effects (model loading, KV cache warming, connection pool initialization). guidellm supports --warmup and --cooldown flags that can be used for this purpose.

Multiple trials: For published results, each scenario should be run multiple times (e.g., 3 trials) and the report should include variance and confidence intervals alongside the percentile metrics. This strengthens credibility and reveals whether observed differences are statistically significant.

Batch workload

• 3 concurrent batch jobs with 1000 requests each, submitted simultaneously before interactive traffic begins.
• All jobs use the same model as interactive traffic.
• Each job has a different completion window SLO to demonstrate SLO-deadline ordering:
  • Job A: 1000 requests, completion_window: 30m (tight)
  • Job B: 1000 requests, completion_window: 2h (moderate)
  • Job C: 1000 requests, completion_window: 24h (relaxed)
• Expected result (gated scenarios): Job A completes first (its requests get priority via SLO-deadline ordering), then B, then C — and all three meet their windows. Under ungated/no-batch-gateway scenarios, Job A is likely to miss its tight window.
• Completion window values should be calibrated during initial runs so that the tight SLO is achievable with gating but not without.

Run duration and survival: The completion_window: 24h SLO on Job C means the system must sustain operation for potentially long periods. The benchmark should verify that a full multi-scenario run completes without infrastructure failures (OOM, connection drops, log rotation issues). For CI, a scaled-down profile with shorter completion windows (e.g., 5m / 15m / 1h) should be used.

Prompt characteristics

Input/output sequence lengths

Existing llm-d benchmarks use statistical distributions (lognormal, normal, uniform) for input sequence length (ISL) and output sequence length (OSL), not fixed token counts. The benchmark should align with this practice to produce representative results.

The canonical profile should define ISL/OSL distributions. Recommended starting point (based on existing llm-d benchmark configurations):

Parameter	Distribution	Mean	Std Dev	Min	Max
ISL (input sequence length)	lognormal	1500	1200	10	4096
OSL (output sequence length)	lognormal	500	400	10	2048

The profile file (benchmarks/profiles/default.yaml) should support alternative distribution types (normal, uniform) and parameter sets for different workload shapes (e.g., document summarization = long ISL / short OSL, text classification = short ISL / very short OSL, conversation replay = mixed).

Data generation approach

The current implementation uses Faker to generate random text of configurable token length. This is sufficient for initial benchmarks focused on dispatch behavior, but the benchmark should support pluggable data sources for workload realism:

1. Synthetic (Faker) — Random text, fastest to generate, fully reproducible. Default for dispatch-focused scenarios.
2. Synthetic with distributions (inference-perf) — Uses the same synthetic generation as existing llm-d benchmarks (ghcr.io/llm-d/llm-d-benchmark), with statistical ISL/OSL distributions. Preferred for ecosystem alignment.
3. Real datasets — HuggingFace datasets (e.g., ShareGPT, MMLU) for production-representative workloads. Recommended for published results and larger-model validation.

Same prompt distribution for both interactive and batch to isolate the dispatch strategy as the variable.

Conversation replay (follow-on)

Existing llm-d benchmarks support conversation replay workloads — multi-turn exchanges with varying ISL/OSL per turn, drawn from real conversation traces (e.g., ShareGPT). This is not needed for the initial dispatch-focused benchmark, but should be supported as a data generation mode in a later PR for production-representative workload validation.

System prompts for prefix-cache evaluation

Batch requests should use a set of distinct system prompts shared across many requests in the batch. The batch processor sorts requests by system-prompt hash (FNV-32a) before dispatch, grouping requests with the same system prompt together. This enables vLLM's prefix cache to reuse the cached KV state for the shared prefix, reducing TTFT for subsequent requests in the same group.

Existing llm-d benchmarks use significantly larger shared-prefix configurations than the initial spec:

Parameter	Initial spec	llm-d ecosystem range	Recommended
Number of groups	3–5	32–600	32–64 (start), scale up in follow-on runs
System prompt length	~50 tokens	2048–6000 tokens	2048 tokens (start)

Comparing scenario 1 (no batch-gateway — prompts sent in random order via guidellm) to scenario 2 (ungated batch-gateway — prompts sorted by system-prompt hash) isolates the efficiency gain from this sorting, since both scenarios are otherwise unthrottled.

Metrics

Interactive workload quality

Metric	Granularity	Purpose
TTFT (time to first token)	p50, p95, p99 per phase	Primary latency indicator — should not degrade during bursts when batch is active
ITL (inter-token latency)	p50, p95, p99 per phase	Decode-phase quality — measures interference with streaming output
TPOT (time per output token)	p50, p95, p99 per phase	Standard llm-d metric — normalized decode latency per token
Request latency (end-to-end)	p50, p95 per phase	Overall request duration
Throughput	req/s per phase	Should remain stable across scenarios during burst phases
Error rate	per phase	429s, 5xx, timeouts — ungated scenarios may show elevated errors

Batch workload effectiveness (scenarios 2–6)

Metric	Purpose
Per-job completion timeline	Completed requests over time per job (A/B/C). Scenario 3 (admission control + AIMD): sawtooth — AIMD backs off on 429s, recovers during idle. Scenario 4 (flow control + AIMD): staircase — progress during idle, slowed during burst as priority ordering favors interactive. Scenario 6 (low concurrency): steady low slope — fixed concurrency, no adaptive behavior. All should show Job A (tight SLO) completing first
Per-job completion time	Wall-clock time to finish each batch job — demonstrates SLO-deadline ordering (A before B before C)
SLO satisfaction	Whether each job completes within its configured completion window (30m / 2h / 24h)
Batch throughput	Requests completed per second (averaged over idle phases)
Batch TTFT	TTFT p50/p95 for batch requests — scenario 2 should show lower TTFT than scenario 1 due to system-prompt sorting enabling prefix cache hits

Batch-equivalent workload (scenario 1 only)

In scenario 1 there are no batch jobs — the "batch" workload is a second guidellm instance (guidellm-batch) sending the same prompts as regular interactive requests. Batch-gateway metrics (completion timeline, SLO satisfaction, per-job ordering) do not apply. Instead, the batch-equivalent workload is measured from guidellm's own output:

Metric	Purpose
Completion count over time	Total requests completed by the batch guidellm instance over time — comparable to the batch completion timeline in other scenarios
Total completion time	Wall-clock time for the batch guidellm instance to finish all requests
Throughput	Requests completed per second
TTFT	TTFT p50/p95 — baseline for comparison with scenario 2 (prefix-cache sorting effect)

This asymmetry is intentional: scenario 1 shows what happens without any batch-gateway features (no SLO prioritization, no system-prompt sorting, no job management).

Infrastructure utilization

Metric	Source	Purpose
GPU utilization	`vllm:gpu_cache_usage_perc`	Shows capacity utilization across phases
Running requests	`vllm:num_requests_running`	Correlates with burst/idle phases and batch dispatch
Waiting requests	`vllm:num_requests_waiting`	Queue pressure indicator
AIMD concurrency limit	`batch_processor_aimd_concurrency_limit`	Scenarios 3/4: shows AIMD sawtooth pattern (S3 active via 429s; S4 flat without 429 source)
AIMD decrease/increase events	`batch_processor_aimd_decreases_total`, `batch_processor_aimd_increases_total`	Scenarios 3/4: shows backoff/recovery dynamics
llm-d Router admission/FC saturation	`inference_extension_flow_control_pool_saturation`	Scenarios 3/4: shows saturation level driving admission (S3) or dispatch ordering (S4)
Router flow control queue size	`inference_extension_flow_control_queue_size`	Scenario 4: batch band queue depth
Dispatch budget	async-processor gate metric	Scenario 5: shows gate open/close pattern

Infrastructure

Hardware

The benchmark should be model/GPU-agnostic with configurable parameters. Documented starting configuration:

• Model: Qwen/Qwen3-8B (fits on 1 GPU, large enough to saturate under combined load)
• GPU: 1× NVIDIA A100 (or equivalent)
• `max-model-len`: 4096

Other tested configurations should be documented as they are validated.

Scaling path: The initial Qwen3-8B / 1×A100 configuration validates dispatch behavior at small scale. Downstream validation should include larger models (e.g., 70B+ on multi-GPU) and multi-node deployments to verify that gated dispatch benefits hold under production-scale conditions. The benchmark framework should be parameterized to support this without script changes.

Software stack

• Kubernetes cluster with GPU nodes
• llm-d Router (EPP) with Istio gateway
• vLLM model server
• Prometheus for metrics collection
• Redis + PostgreSQL for batch-gateway
• guidellm for interactive traffic generation
• For scenarios 3 and 4: `InferenceObjective` CRDs (batch-sheddable priority -1, interactive-default priority 100) + saturation detector configured on the llm-d Router
• For scenario 3: llm-d Router admission controller (no flow control feature gate) — rejects sheddable requests at saturation ≥ 1.0
• For scenario 4: llm-d Router flow control feature gate + `EndpointPickerConfig` with priority bands (see Flow Control Setup Guide)
• For scenario 5: llm-d-async processor

Deployment

Automated via setup.sh / teardown.sh scripts that deploy the full stack per scenario. Each scenario runs in its own namespace for clean setup/teardown isolation — scenarios are executed sequentially (not in parallel) since they share GPU infrastructure and concurrent execution would cause interference. The benchmark guide documents what the scripts do, so users can adapt for custom environments.

Artifact sourcing

The setup scripts should deploy from published artifacts with pinned versions by default — not from local git checkouts. This ensures benchmark results are reproducible across operators and CI runs, and avoids confounding results with whatever commit happens to be checked out locally.

Components that should be version-pinned:

• Router chart — pin to a released OCI version (e.g., oci://ghcr.io/llm-d/llm-d-router-gateway --version 0.3.0)
• Router values / guides — pin to a tagged version from llm-d
• batch-gateway chart — pin to a released version or explicit --set-string image.tag=...
• vLLM image tag — pin in the Kustomize overlay
• Model revision — pin to a specific HuggingFace commit SHA (e.g., --revision abc123 in vLLM args). Model authors can update weights, tokenizer configs, or architecture at any time; without pinning, Qwen/Qwen3-8B pulls the latest main snapshot and results aren't comparable across runs

Redis and PostgreSQL do not need pinning — they are infrastructure dependencies whose version differences won't affect benchmark results.

Local repo overrides (LLM_D_REPO, ROUTER_REPO env vars) should be supported as an opt-in for development, but the default path should require no extra repos beyond batch-gateway itself.

Parameter reproducibility

Benchmark parameters (prompt token length, system prompt count, seed, burst/idle rates, phase durations, cycles, batch size, job count) must be versioned and recorded to ensure results are comparable across runs.

Canonical profiles: A checked-in config file (e.g., benchmarks/profiles/default.yaml) should define the default parameter set that both generate_prompts.py and benchmark.py read. CLI flags may override for development, but the default path should produce comparable runs without requiring any flags.

Run metadata: Each benchmark run should emit a machine-readable run-metadata.json alongside results, recording:

• The profile name and all parameter values used
• Stack versions (batch-gateway, router, vLLM) and model revision (HuggingFace commit SHA)
• Timestamp

This enables programmatic comparison between runs and automated drift detection. The HTML report's Configuration table should also be extended to include all parameters (currently missing prompt_tokens, num_system_prompts, seed, and stack versions).

Report

An auto-generated HTML report (self-contained, Chart.js) containing:

1. Summary table — one row per scenario, designed to be directly usable in blog posts and conference talks:

Scenario	Interactive TTFT at peak (p50 / p95 / p99)	Interactive ITL at peak (p50 / p95 / p99)	Batch idle throughput (req/s)	Batch SLO completion (actual / target)	Batch TTFT p50
0. Interactive only	X / X / X ms	X / X / X ms	—	—	—
1. No batch-gateway	X / X / X ms	X / X / X ms	X req/s	N/A	X ms
2. Ungated batch	X / X / X ms	X / X / X ms	X req/s	A: ✗ 45m/30m B: ✗ 3h/2h C: ✓ 5h/24h	X ms
3. Admission control + AIMD	X / X / X ms	X / X / X ms	X req/s	A: ✓ 20m/30m B: ✓ 50m/2h C: ✓ 2h/24h	X ms
4. Flow control + AIMD	X / X / X ms	X / X / X ms	X req/s	A: ✓ 15m/30m B: ✓ 45m/2h C: ✓ 2h/24h	X ms
5. Async processor	X / X / X ms	X / X / X ms	X req/s	A: ✓ 18m/30m B: ✓ 48m/2h C: ✓ 2h/24h	X ms
6. Low concurrency	X / X / X ms	X / X / X ms	X req/s	A: ? Xm/30m B: ? Xm/2h C: ? Xh/24h	X ms

2. Detailed per-phase breakdown — per-scenario, per-phase (burst/idle) table with full metric set for deeper analysis.
3. Narrative conclusion — auto-generated based on metric comparisons.
4. Batch completion timeline — line chart with burst/idle phase bands overlaid. All scenarios on one chart for direct comparison.
5. Interactive latency comparison — bar charts for TTFT and ITL (p50/p95/p99) grouped by phase and scenario.
6. Error rate comparison — incomplete/failed request rates per scenario.
7. Infrastructure metrics — GPU utilization and running request count over time, per scenario.
8. AIMD dynamics (scenarios 3/4) — concurrency limit over time showing the sawtooth pattern (S3: active via 429s; S4: expected flat without 429 source).
9. llm-d Router dynamics (scenarios 3/4) — saturation level over time; scenario 3 shows admission rejection at saturation ≥ 1.0; scenario 4 shows flow control queue size and dispatch ordering behavior.

Machine-readable results

In addition to the HTML report, each benchmark run should emit a structured JSON results file (e.g., results.json) containing all metrics in a format compatible with inference-perf output. This enables:

• Programmatic comparison across runs (CI regression detection, A/B analysis)
• Cross-referencing with other llm-d benchmark results that use inference-perf
• Ingestion into dashboards or analysis notebooks without HTML scraping

The JSON schema should include: scenario ID, all latency percentiles (TTFT, ITL, TPOT), throughput, error rates, batch completion timelines, and the full run-metadata.json contents (profile, versions, timestamp).

Ecosystem alignment

This benchmark should align with the existing llm-d benchmarking ecosystem to ensure consistent methodology and comparable results across the project.

inference-perf harness

Most llm-d benchmark guides use inference-perf (ghcr.io/llm-d/llm-d-benchmark) as the benchmark harness. It supports:

• Random synthetic, shared prefix, and conversation replay workloads
• Statistical ISL/OSL distributions (lognormal, normal, uniform)
• Rate sweeps for throughput-latency characterization
• Standard output metrics (TTFT, TPOT, ITL, throughput)

The batch-gateway benchmark uses guidellm for interactive traffic (which is appropriate — guidellm's burst/idle pattern is central to the methodology). However, the data generation and workload shape definitions should be compatible with inference-perf configurations where possible, so results can be cross-referenced with other llm-d benchmarks.

Relevant existing benchmarks

Guide	ISL	OSL	Workload type
Optimized baseline	128–8192 (uniform)	10–2048 (uniform)	Random synthetic
PD disaggregation	5000 (fixed)	250 (fixed)	Long-context
Prefix cache routing	6000 system prompt + 1200 question	—	Shared prefix
Predicted latency routing	lognormal(1500, 1200)	lognormal(800, 600)	Distribution-based
Workload autoscaling	normal(4096, 2048)	10–2048	High-variance
Router benchmark	ShareGPT dataset	—	Real dataset

MLPerf Offline reference

Scenario 2 (ungated batch with maximum concurrency) is functionally similar to the MLPerf Inference offline scenario — all samples are available upfront and the system processes them as fast as possible. While the batch-gateway benchmark focuses on the OpenAI Batch API interface (not MLPerf compliance), scenario 2 throughput numbers can serve as an upper-bound reference point comparable to MLPerf offline results.

Existing work

@evacchi has built significant benchmarking infrastructure in evacchi#1 covering the sync vs gated-async comparison. This includes:

• `benchmark.py` — Python orchestrator (1069 lines) handling cleanup, batch submission, guidellm traffic, monitoring, result collection, and HTML report generation
• `setup.sh` / `teardown.sh` — Full-stack Kubernetes deployment and cleanup
• Helm values for sync and async dispatch modes
• Kustomize overlay for vLLM (Qwen3-8B)
• guidellm burst/idle Job templates
• Multiple benchmark runs with results and reports

This work is the starting point. The implementation tasks below extend it to cover all scenarios, add the "no batch-gateway" baseline, add AIMD and flow control metrics, and produce a unified comparison report.

Directory structure

All benchmark machinery lives in benchmarks/ at the repo root:

benchmarks/
├── benchmark.py                  # orchestrator (scenario selection, metric collection, report generation)
├── generate_prompts.py           # Faker-based prompt generation with system-prompt diversity
├── setup.sh                      # deploy full stack per scenario
├── teardown.sh                   # cleanup
├── README.md                     # methodology, usage, interpretation
├── helm-values/
│   ├── scenario-0-interactive-only.yaml
│   ├── scenario-1-no-batch-gateway.yaml
│   ├── scenario-2-ungated.yaml
│   ├── scenario-3-admission-control-aimd.yaml
│   ├── scenario-3-admission-control-overlay-router.yaml
│   ├── scenario-4-aimd-flow-control.yaml
│   ├── scenario-4-flow-control-overlay-router.yaml
│   ├── scenario-5-async.yaml
│   └── scenario-6-low-concurrency.yaml
├── manifests/
│   ├── guidellm-interactive.yaml # burst/idle interactive traffic Job
│   ├── guidellm-batch.yaml       # batch-equivalent traffic for scenario 1
│   └── vllm/                     # Kustomize overlay for vLLM
└── results/                      # gitignored — local run output
    └── .gitkeep

Local testing

The benchmark orchestration, scenario switching, metric collection, and report generation can be tested locally on a Kind cluster using llm-d-inference-sim (ghcr.io/llm-d/llm-d-inference-sim:latest) instead of real vLLM. The simulator supports configurable TTFT, inter-token latency, and failure injection (429 rate-limiting at configurable rates) — the same simulator that make dev-deploy and the e2e tests already use. Metrics won't reflect real GPU performance, but the full pipeline can be validated end-to-end without a GPU.

Implementation tasks

PR 1: Port infrastructure

• Port benchmark infrastructure — Cherry-pick the benchmark scripts (benchmark.py, setup.sh, teardown.sh), Helm values, Kustomize overlays, and guidellm Job templates from evacchi/batch-gateway#1 into a new PR on the main repo under benchmarks/. Exclude result files and async-processor dependencies (that PR depends on the async integration which hasn't landed yet). Restructure into the directory layout defined above
• Prompt generation — Implement generate_prompts.py: Faker-based random text of configurable token length, with a small set of distinct system prompts (3–5) shared across many requests to enable prefix-cache evaluation
• Benchmark README — benchmarks/README.md covering methodology, prerequisites, usage instructions, scenario descriptions, and how to interpret results

PR 1.5: Kind + inference-sim e2e path + workload shape alignment

Enable full-pipeline validation (setup → benchmark → collect → report) on a local Kind cluster using llm-d-inference-sim instead of real vLLM. This unblocks testing for all subsequent PRs without requiring CoreWeave cluster access. Also upgrades the workload generation to align with llm-d ecosystem benchmark practices (ISL/OSL distributions, scaled-up shared prefix, TPOT metric).

• Sim mode in setup.sh — Add a MODE=sim (or similar) flag that substitutes llm-d-inference-sim for vLLM and skips components that aren't needed in sim mode (router, Istio gateway). Add simplified Helm values for the sim path (no GPU resources, no model download)
• ISL/OSL distributions in generate_prompts.py — Replace fixed prompt_tokens with configurable ISL/OSL distributions (lognormal, normal, uniform). Update profiles/default.yaml with the recommended starting parameters (ISL: lognormal mean 1500 stdev 1200, OSL: lognormal mean 500 stdev 400)
• Scale up shared prefix config — Increase system prompt groups from 3–5 to 32–64, and system prompt length from ~50 tokens to 2048 tokens, matching the llm-d ecosystem range
• Add TPOT metric — Collect TPOT (time per output token) at p50/p95/p99 in benchmark.py alongside TTFT and ITL
• Local e2e target — Add a make benchmark-local (or similar) Makefile target that runs the full pipeline on Kind with inference-sim
• Validate — End-to-end run on Kind: setup, prompt generation, benchmark execution, result collection, HTML report generation

PR 2: Baseline scenarios (0–2)

• Add scenario 0 (interactive-only baseline) — Helm values, setup/teardown, and orchestrator support for deploying just vLLM + llm-d Router + guidellm with the burst/idle interactive traffic pattern, no batch workload of any kind. This establishes the ideal interactive latency baseline that all other scenarios are compared against
• Add scenario 1 (no batch-gateway baseline) — Helm values, setup/teardown, and orchestrator support for adding a second guidellm instance (guidellm-batch) that sends batch-equivalent prompts as regular interactive requests to the inference gateway. Same prompt distribution and volume as the batch workload
• Add scenario 2 (ungated batch) — Helm values for sync dispatch with aggressive concurrency (global: 200, perEndpoint: 100, aimd.enabled: false). Edoardo's processor-sync-aggressive-values.yaml is a starting point
• Metrics and report (baseline) — Extend benchmark.py to: (1) collect TTFT/ITL/TPOT at p50/p95/p99, (2) query Prometheus for GPU utilization and running/waiting requests, (3) generate HTML report with scenarios 0–2 side-by-side. Make model name, GPU count, max-model-len, burst rate, and phase duration configurable via CLI flags/env vars. Configure guidellm with --warmup to exclude the first burst/idle cycle from results (avoids cold-start artifacts)
• Machine-readable results — Emit a structured results.json per run containing all metrics (latency percentiles, throughput, error rates, batch timelines) and run metadata. Schema should be compatible with inference-perf output for cross-referencing with other llm-d benchmarks
• Pluggable data generation — Extend generate_prompts.py to support data sources beyond Faker: (1) inference-perf standard synthetic generation with ISL/OSL distributions (aligns with llm-d ecosystem methodology), (2) real datasets from HuggingFace (e.g., ShareGPT). Published results should use at least inference-perf synthetic; real dataset results strengthen credibility for blog posts and talks
• Rate sweep mode — Add a rate-sweep mode to benchmark.py that runs each scenario across a range of request rates (e.g., 1–30 rps) to produce throughput-latency curves. This shows where gating starts protecting interactive latency — essential for published results beyond a single data point. Include an auto-calibration step: a short preliminary sweep to discover the saturating rate for the current hardware/model, which is then used as the burst rate for the main benchmark run
• Multiple trial runs — Support running each scenario N times (e.g., --trials 3) and report variance/confidence intervals alongside percentile metrics. Strengthens credibility for published results and reveals whether observed differences are statistically significant

PR 3: Gated scenarios (3–4)

• Add scenario 3 (AIMD only) — Helm values for sync dispatch with AIMD enabled (default parameters: min: 5, backoffFactor: 0.5, additiveIncrease: 1), llm-d Router flow control disabled
• Add scenario 4 (AIMD + llm-d Router flow control) — Same processor config as scenario 3, plus: enable llm-d Router flowControl feature gate, deploy EndpointPickerConfig with priority bands (interactive: 100, batch: -1 sheddable), deploy InferenceObjective CRDs, configure processor with inferenceObjective: "batch-sheddable". See Flow Control Setup Guide
• Metrics and report (gated) — Extend metric collection to include AIMD metrics (batch_processor_aimd_concurrency_limit, decrease/increase counters) for scenarios 3/4 and llm-d Router flow control metrics (inference_extension_flow_control_pool_saturation, queue size) for scenario 4. Extend HTML report with all scenarios (0–4) side-by-side plus AIMD and flow control dynamics charts

PR 3.1: Refined gated scenarios + S6

Refine scenarios 3 and 4 based on benchmark findings (Runs 7–8 analysis), and add scenario 6. The original S3 (AIMD-only) could not generate 429s because it lacked InferenceObjective CRDs and the inferenceObjective header. The original S4 used utilization-detector which is less reliable than concurrency-detector for the priority dispatch use case.

• Refine scenario 3 (admission control + AIMD) — Replace the original AIMD-only config with admission control + AIMD: processor configured with globalInferenceGateway + inferenceObjective: "batch-sheddable". New router overlay with concurrency-detector saturation detector but NO flowControl feature gate — uses the llm-d Router's admission controller which rejects sheddable requests (priority < 0) at saturation ≥ 1.0. setup.sh extended to deploy InferenceObjective CRDs and router overlay for S3
• Refine scenario 4 (flow control + AIMD) — Update router overlay: switch saturation detector from utilization-detector to concurrency-detector with generous maxConcurrency for free-running priority dispatch ordering. Processor config unchanged (already has inferenceObjective: "batch-sheddable")
• Add scenario 6 (low concurrency) — New Helm values for sync dispatch with fixed low concurrency (perEndpoint: 20, aimd.enabled: false). No router-side protection — processor concurrency cap is the only gating mechanism
• Update setup.sh — Extend range validation (0–7), case statement, CRD deployment, router overlay application, and EPP routing for S3, S6

PR 4: GPU benchmark automation + summary table

Reduce the GPU cluster benchmark procedure to a single benchmark.py invocation that runs all scenarios end-to-end (setup → benchmark → teardown per scenario) and produces a unified cross-scenario comparison report with the full summary table.

• ServiceMonitor automation — Have setup.sh (GPU mode) create the batch-gateway processor metrics Service and ServiceMonitor automatically. Parameterize the Prometheus discovery label via env var (e.g. PROMETHEUS_RELEASE, default llmd-kube-prometheus-stack) since it varies across clusters
• Port-forward automation — Have benchmark.py manage the Prometheus port-forward lifecycle internally: start a background kubectl port-forward before metric collection, tear it down after. Accept --prometheus-namespace and --prometheus-service flags instead of requiring the user to set up a manual port-forward. Fall back to PROMETHEUS_URL if provided (for clusters with direct Prometheus access)
• EPP image tag parameterization — Extract the hardcoded v0.8.0 EPP image tag in the Router local-repo path to an env var (e.g. ROUTER_EPP_TAG) with a default, consistent with the existing GIE_VERSION / ROUTER_CHART_VERSION pattern
• Managed orchestration in benchmark.py — Add a --managed flag (or equivalent) that makes benchmark.py call setup.sh before and teardown after each scenario within its own multi-scenario loop. This keeps all ScenarioResult objects in a single process, so generate_html_report is called once with all results and produces the cross-scenario summary table, comparison charts, and unified results.json naturally. The alternative (a shell wrapper that calls benchmark.py per scenario separately) would require building a new report-merging mode to achieve what the single-process approach gets for free
• make benchmark-gpu target — Add a Makefile target that invokes benchmark.py --managed --scenarios 0 2 3 4 (plus context, namespace, and Prometheus flags) for GPU cluster runs, analogous to the existing make benchmark-local for Kind
• Full summary table — Update generate_html_report to produce the one-row-per-scenario summary table as specified in the Report section above. The current report has a per-phase breakdown but is missing the top-level summary with: interactive TTFT p99 burst, interactive ITL p99 burst, batch idle throughput (req/s), batch SLO completion (actual vs target per job), and batch TTFT p50. All of this data is already collected or derivable from existing ScenarioResult fields — this is a presentation change, not a metrics collection change
• Replace ConfigMap with PVC for prompt data — The batch JSONL input file is currently stored in a Kubernetes ConfigMap ({name}-data), which has a hard 1MB size limit. With large ISL (256+ tokens) needed to stress TTFT and 1000 requests per job, the JSONL exceeds this limit. Replace the ConfigMap-based delivery with PVC-based storage — write the JSONL to a PersistentVolumeClaim (e.g., the existing benchmark-results PVC or a dedicated one) and mount it into the batch-submit pod. This removes the size constraint and unblocks realistic prompt sizes

PR 5: Report completeness

Enhance the HTML report to cover all items specified in the Report section above. Depends on having real GPU benchmark data from the "Run and validate" step to validate the charts and metrics.

• Detailed per-phase breakdown — Already exists; verify it renders correctly with real GPU data
• Narrative conclusion — Auto-generate a text summary based on metric comparisons (e.g., "Scenario 4 protected interactive TTFT within X% of the baseline while completing all batch SLOs")
• Batch completion timeline with phase bands — Overlay burst/idle phase bands on the existing timeline line chart so readers can see how batch progress correlates with interactive load phases
• ITL bar chart — Add ITL (inter-token latency) p50/p95/p99 bar chart alongside the existing TTFT and TPOT charts
• Error rate comparison chart — Dedicated chart for incomplete/failed request rates per scenario (currently only shown in the table)
• Infrastructure metrics charts — GPU utilization (vllm:gpu_cache_usage_perc) and running request count (vllm:num_requests_running) over time, per scenario. Requires Prometheus queries against vLLM metrics
• Flow control queue size chart — Add Router flow control queue size (inference_extension_flow_control_queue_size) alongside the existing saturation chart for scenario 4

PR 6: Sim mode flow control + admission control support

Enable scenarios 3 and 4 validation on a local Kind cluster without a GPU, reusing the approach from make dev-deploy-gie (which already deploys per-model EPP with flow control on Kind using llm-d-inference-sim).

• Scenario 4 sim mode — Extend setup.sh sim mode (MODE=sim) to deploy the Router/EPP with flow control when SCENARIO=4: per-model EPP standalone deployment, EndpointPickerConfig with flowControl feature gate, InferenceObjective CRDs (interactive-default priority 100, batch-sheddable priority -1). Follow the existing pattern in scripts/dev-deploy.sh (ENABLE_GIE). This enables BENCHMARK_SCENARIO=4 make benchmark-local to work end-to-end on Kind
• Summary table SLO completion format — The summary table's "Batch SLO completion" column currently shows completed/total request counts per job instead of the per-job actual-time/target-time format with ✓/✗ indicators specified in the Report section (e.g., A: ✓ 20m/30m). Update to show actual completion time vs target window with pass/fail indicators
• Infrastructure metrics: requests_waiting chart — The vllm:num_requests_waiting series is collected via Prometheus but not charted. Add it as a third infrastructure metrics chart alongside GPU cache usage and requests running — it serves as the queue pressure indicator described in the Metrics section

Run and validate

• Run and validate — Execute the full benchmark on a GPU cluster, validate results, iterate on parameters

PR 7: Benchmarks README

Update benchmarks/README.md to document the full benchmark procedure for both sim (Kind) and GPU cluster modes, covering everything a new contributor needs to run benchmarks end-to-end.

• Sim mode (Kind) walkthrough — Document the make benchmark-local flow: prerequisites (Kind, Docker), what it deploys, how to inspect results, and how to customize scenarios/parameters
• GPU cluster walkthrough — Document the make benchmark-gpu flow (after PR 4): prerequisites (cluster access, GHCR credentials, Prometheus), env vars, step-by-step instructions, and common troubleshooting (RBAC, ConfigMap size limits, ServiceMonitor discovery labels)
• Scenario reference — Brief description of each scenario (0–6), what it tests, and how to interpret results — linking back to the detailed spec in this issue
• Configuration reference — Document all env vars and CLI flags accepted by setup.sh, benchmark.py, generate_prompts.py, and teardown.sh

PR 8: Async scenario (5)

Blocked on batch-gateway ↔ async-processor integration merge.

• Add scenario 5 (gated async) — Helm values for async dispatch mode with prometheus-budget gate. Extend setup/teardown and orchestrator to deploy and configure async-processor. Collect dispatch budget metrics (gate open/close state, budget values over time). Extend HTML report with scenario 5 on all comparison charts plus dispatch budget dynamics plot

PR 9: Flow control setup guide update

Update docs/guides/flow-control-setup.md to reflect the refined benchmark configurations and document the admission control path.

• Saturation detector recommendation — Update the guide's EndpointPickerConfig to recommend concurrency-detector (with appropriate maxConcurrency) as an alternative to utilization-detector, based on benchmark findings. Document when each is appropriate
• Admission control section — Add a new section covering the "without flow control" setup: admission controller + AIMD. Covers InferenceObjective CRDs for priority assignment, saturation detector configuration, and how the admission controller rejects sheddable requests at saturation ≥ 1.0 to generate 429s for AIMD. Explains when to use this simpler path vs full flow control
• Priority dispatch ordering vs saturation-based rejection — Clarify the distinction between these two mechanisms in the "How Flow Control Works" section. Dispatch ordering (passive, works in current releases) vs per-band saturation ceilings (requires priority-holdback-policy, not yet released)

Documentation

• Results archive — Decide on a strategy for archiving benchmark results (in-repo, external, CI artifact)