Skip to content

validation.md

Latest commit

 

History

History
319 lines (253 loc) · 16 KB

validation.md

File metadata and controls

319 lines (253 loc) · 16 KB
Raw

Validation log

2026-07-22 — DGX Spark M0

Host facts:

  • hostname: spark-7820
  • architecture: aarch64
  • Python: 3.12.3
  • GPU: NVIDIA GB10
  • driver: 580.159.03

Checks completed on the DGX Spark:

  • 11/11 Python unit tests passed after the gateway and permit-pool tests were added;
  • both shell scripts passed bash -n;
  • FCFS and weighted-fair simulations replayed the same 300-request, seed-7, 4-request/s trace.

Simulator summaries:

Policy SLO goodput Jain attainment Min tenant attainment p95 TTFT
FCFS 9.0% 0.544 2.90% 14,191.6 ms
weighted fair 24.67% 0.609 4.08% 16,733.2 ms

Interpretation: the result is a sanity check for policy ordering and metric accounting. The simulator is a single-worker approximation and does not model continuous batching, prefill/decode overlap, KV pressure, or GPU behavior. It is not evidence that weighted fairness improves vLLM performance.

The GPU baseline was initially blocked because the SSH user foo could not access /var/run/docker.sock. That access was corrected before the next validation step.

2026-07-22 — DGX Spark GPU smoke and M1 low-rate baseline

Docker access was enabled for foo and GPU passthrough was verified from a container. The first serving smoke test used cached Qwen/Qwen3-0.6B; this model is intentionally classified as an M0 API/metrics check because it is not in NVIDIA's vLLM-on-Spark support matrix.

The M1 baseline uses:

  • image tag: nvcr.io/nvidia/vllm:26.05.post1-py3;
  • image digest: sha256:94e21552f644e0c1627464ba89d2f7a4ce7442e196f72afa0bb5d7fba23cbb03;
  • vLLM: 0.21.0+2325b6f0 from NVIDIA release 26.05.post1;
  • model: nvidia/Qwen3-8B-FP8 at revision 2cebc4c89e25abc17668c81b01dceaf3d8b914d56;
  • ModelOpt FP8 weights, FP8 E4M3 KV cache, FlashInfer scaled-matrix kernels and attention;
  • maximum model length 8192, maximum sequences 32, GPU-memory utilization 0.80, prefix caching enabled, and vLLM generation defaults;
  • 256 input tokens, 64 forced output tokens, 0.5 requests/s Poisson arrivals, 50 measured requests plus 5 warmups, seed 7, and temperature 0.

Declared goodput SLOs were TTFT <= 1000 ms, TPOT <= 100 ms, and E2E <= 15000 ms.

Metric Corrected low-rate run
Completed / failed 50 / 0
Request throughput 0.487 req/s
Request goodput 0.487 req/s (100%)
Output throughput 31.18 token/s
Total token throughput 155.89 token/s
p50 / p95 / p99 TTFT 102.8 / 125.2 / 130.6 ms
p50 / p95 / p99 TPOT 39.5 / 40.7 / 40.9 ms
p50 / p95 / p99 E2E 2590.8 / 2651.4 / 2670.2 ms

The first exploratory run omitted an explicit temperature because vLLM 0.21 no longer defaults the benchmark client to greedy sampling. The runner now pins temperature 0 and saves per-request detail; the table above is the corrected run.

Raw artifacts are in results/raw/fp8-low-rate-r2/ and are ignored by Git by default. NVFP4 is reserved for a subsequent paired comparison so quantization does not confound the first scheduler baseline.

2026-07-22 — FP8 saturation sweep

The high-load sweep kept the model, token lengths, seed, sampling settings, and SLOs fixed. Each point used 200 measured requests and 10 warmups.

Offered rate Achieved throughput Goodput Goodput share Peak concurrency p95 TTFT p95 TPOT p95 E2E
8 req/s 7.15 req/s 7.15 req/s 100.0% 38 195.6 ms 49.2 ms 3,242.7 ms
12 req/s 9.87 req/s 9.43 req/s 95.5% 52 983.2 ms 43.2 ms 3,649.6 ms
16 req/s 9.97 req/s 2.59 req/s 26.0% 94 4,295.8 ms 43.2 ms 6,973.8 ms
24 req/s 10.07 req/s 1.61 req/s 16.0% 136 7,559.6 ms 43.1 ms 10,240.8 ms

The service saturates near 10 req/s for this fixed 256-input/64-output workload. TTFT and queueing fail first while decode TPOT stays nearly flat. ElasticServe's first production control target should therefore be overload admission and queue policy, not an attempt to minimize active decode concurrency.

Raw artifacts are in results/raw/fp8-highload-rps*/.

2026-07-22 — M2 FCFS gateway

The FastAPI gateway was built and run on the DGX Spark in front of the same vLLM process. Validation covered readiness, non-buffered SSE streaming, request-ID propagation, the queue-time response header, Prometheus metrics, client disconnect accounting, and the completed-stream path.

An exact low-rate pair used 50 requests, 10 warmups, and the same seed and token lengths through the direct and gateway paths:

Path Throughput / goodput p95 TTFT p95 TPOT p95 E2E
Direct vLLM 0.487 / 0.487 req/s 128.3 ms 40.8 ms 2,648.3 ms
Gateway / 32 0.487 / 0.487 req/s 130.6 ms 40.6 ms 2,655.4 ms

The measured proxy delta was +2.3 ms p95 TTFT and +7.1 ms p95 E2E; TPOT was unchanged within run-to-run noise. All requests met the declared SLOs.

At 16 offered req/s, static permit limits produced:

Path / permits Throughput Goodput Peak client concurrency p95 TTFT p95 TPOT p95 E2E
Direct vLLM 9.97 req/s 2.59 req/s 94 4,295.8 ms 43.2 ms 6,973.8 ms
Gateway / 32 9.93 req/s 2.53 req/s 94 4,383.0 ms 42.8 ms 7,040.4 ms
Gateway / 24 7.88 req/s 0.98 req/s 117 9,110.4 ms 42.0 ms 11,742.9 ms
Gateway / 16 5.54 req/s 0.44 req/s 141 19,307.5 ms 41.6 ms 21,918.0 ms

This falsifies the initial idea that lowering gateway permits alone would improve the overload point. vLLM needs enough active requests to form efficient batches; holding requests outside the engine at 16 or 24 permits sacrifices throughput and makes the end-to-end queue grow faster. The active gateway was restored to 32 permits after the experiment.

Design implication: keep the engine's 32-request batching envelope available. The next controller should use queue delay and estimated service capacity for admission/rejection, while scheduling accepted work into that envelope. It should not reduce permits below the measured batching knee as its primary overload response.

The next implementation added a cancellation-safe queue-wait deadline. With the 32-permit envelope fixed, the same 16 req/s trace produced:

Queue budget Completed Rejected SLO-passing / all offered SLO goodput p95 TTFT p95 E2E
Disabled 200 0 51 / 200 (25.5%) 2.53 req/s 4,383.0 ms 7,040.4 ms
1,000 ms 159 41 111 / 200 (55.5%) 6.89 req/s 1,110.4 ms 3,784.9 ms
750 ms 155 45 155 / 200 (77.5%) 9.78 req/s 868.0 ms 3,551.9 ms
500 ms 152 48 152 / 200 (76.0%) 9.72 req/s 619.1 ms 3,302.1 ms

The 750 ms budget is the best of these single runs for the declared 1,000 ms TTFT SLO: it preserves engine throughput, keeps every completed request within all three SLOs, improves goodput rate by 3.86x, and raises all-offered SLO attainment from 25.5% to 77.5%. The 500 ms budget buys lower latency by rejecting three additional requests; the 1,000 ms budget admits work whose backend time pushes TTFT over the SLO.

Gateway counters exactly matched the 750 ms run: 165 backend responses (10 warmups plus 155 measured requests) and 45 queue-timeout rejections. The active DGX gateway was left at 32 permits and a 750 ms queue budget.

These are single-run tuning results, not confidence intervals. The next evidence gate is at least five repeats of the no-admission and 750 ms policies, followed by mixed prompt/output lengths where one fixed queue budget is unlikely to suffice.

Raw artifacts are in results/raw/gateway-fcfs*/ and results/raw/gateway-admit*/.

2026-07-22 — five-seed paired admission evaluation

The fixed 750 ms budget was selected using seed 7, then evaluated on five new seeds: 11, 23, 37, 53, and 71. Each pair used the same seed, 200 measured requests, 10 warmups, 16 req/s offered load, 256 input tokens, 64 forced output tokens, and 32 gateway permits. The control disabled the queue deadline; the candidate enabled 750 ms. Policy order alternated across pairs.

Seed Control goodput Candidate goodput Control attainment Candidate attainment Candidate rejected Control p95 TTFT Candidate p95 TTFT
11 1.37 req/s 10.23 req/s 16.0% 79.5% 41/200 7,126.2 ms 852.9 ms
23 1.70 req/s 8.09 req/s 18.0% 64.5% 71/200 5,763.4 ms 897.9 ms
37 1.43 req/s 9.68 req/s 16.5% 76.5% 47/200 6,974.7 ms 856.1 ms
53 1.50 req/s 8.79 req/s 16.0% 64.0% 72/200 5,437.4 ms 917.3 ms
71 1.35 req/s 9.62 req/s 16.0% 76.5% 47/200 7,332.5 ms 860.3 ms

Every candidate completion met all declared SLOs. Rejected requests remain in the all-offered denominator above; the control completed every request but served most of them after the TTFT deadline.

Paired median effects, using 20,000 paired bootstrap resamples:

  • goodput increase: 8.25 req/s (95% bootstrap interval 6.39 to 8.86);
  • all-offered SLO-attainment increase: 60.0 percentage points (46.5 to 63.5);
  • p95 TTFT reduction: 6,118.7 ms (4,520.1 to 6,472.2);
  • p95 E2E reduction: 6,285.8 ms (4,221.2 to 6,895.9);
  • median goodput ratio: 6.79x;
  • candidate p95 TTFT: median 860.3 ms, range 852.9 to 917.3 ms;
  • candidate rejection rate: median 23.5%, range 20.5% to 36.0%.

Five pairs are enough to establish repeatability for this fixed workload, but the bootstrap intervals are descriptive rather than evidence of broad workload generalization. Burst shape materially changes rejection rate. The next test must use mixed prompt/output lengths and replace the fixed timeout with a service-time or deadline-risk estimate.

Raw artifacts are in results/raw/admission-repeat-*/. The analysis is reproducible with scripts/analyze_admission_pairs.py.

2026-07-22 — mixed-length pilot and service-cost profile

The first mixed workload sampled input lengths uniformly from 64–960 tokens and requested output lengths uniformly from 32–224 tokens using vLLM's independently configured range ratios. It used seed 101, 200 measured requests, 10 warmups, and 8 req/s offered load.

Policy Completed Rejected SLO-passing / all offered Goodput p95 TTFT p95 TPOT p95 E2E
Deadline disabled 200 0 39 / 200 (19.5%) 0.78 req/s 14,227.5 ms 58.8 ms 22,865.4 ms
750 ms queue deadline 156 44 156 / 200 (78.0%) 4.50 req/s 849.6 ms 45.5 ms 9,834.4 ms

The fixed deadline therefore generalized to this one broader workload, improving goodput by 5.76x and all-offered attainment by 58.5 percentage points. It is only a pilot, not a repeated mixed-workload result.

Mapping rejected request positions back to the identical control trace showed a small size skew: accepted requests averaged 621 requested input-plus-output tokens, versus 651 for rejected requests. A timeout does not directly inspect request size, but long admitted generations consume capacity and affect unrelated requests behind them. Cost-aware queue ordering therefore remains necessary for the multi-tenant experiment.

A separate direct-backend profile used maximum concurrency 1, five measured requests and two warmups at four fixed length points:

Input / output tokens p95 TTFT p95 TPOT p95 E2E
64 / 32 53.9 ms 40.2 ms 1,299.2 ms
256 / 64 65.7 ms 40.8 ms 2,632.2 ms
512 / 128 68.3 ms 41.5 ms 5,339.5 ms
960 / 224 88.0 ms 41.5 ms 9,347.3 ms

The fitted low-concurrency relative-cost model is:

  • TTFT = 52.494 ms + 0.028640 ms per input token;
  • TPOT = 41.063 ms per requested output token.

This is a queue-ranking cost estimate, not an overloaded latency predictor: continuous batching changes absolute wall time under concurrency. The calibrated model is stored in config/service-cost-fp8.json; WeightedFairScheduler can now use it instead of treating input and output tokens as equally expensive.

Raw artifacts are in results/raw/mixed-pilot-*/ and results/raw/service-profile-*/. The next implementation gate is a custom client that sends exact request token counts and tenant/SLO metadata to the live gateway, followed by paired FCFS-versus-cost-aware weighted-fair tests.

2026-08-18 — M3 paired FCFS versus weighted-fair queue ordering

The gateway gained QUEUE_POLICY=wfq: a cancellation-safe start-time-fair permit queue whose virtual finish tags advance each tenant's clock by estimated cost / weight, with cost from the calibrated prefill/decode model in config/service-cost-fp8.json and weights/SLOs from config/tenants.example.toml (gold 4.0, standard 2.0, batch 1.0). A new tenant-aware open-loop client (elasticserve-loadgen) sends exact token-id prompts with tenant and token-count headers, so gateway cost estimates use exact counts, never tokenizer guesses. All 24 unit tests passed, and the loadgen path was smoke-tested against the live gateway before measurement.

Experiment controls:

  • same pinned vLLM backend, image, and model as all prior entries;
  • both policies behind the identical admission rule: 32 permits, 256-deep queue, fixed 750 ms queue-wait budget;
  • five paired seeds (11, 23, 37, 53, 71), 200 measured requests plus 10 warmups each, 8 req/s offered Poisson arrivals, mixed 64–960 input and 32–224 output tokens, tenant mix gold 20% / standard 50% / batch 30%;
  • policy order alternated across seeds; each policy run used a distinct prompt salt so replayed traces keep identical arrivals, tenants, and lengths while token identity differs (this vLLM build exposes no reset_prefix_cache endpoint, so salting prevents cross-run prefix-cache hits);
  • SLO attainment is per-tenant (gold 600/5000 ms, standard 1200/9000 ms, batch 5000/30000 ms TTFT/E2E), with rejections kept in the all-offered denominator.
Seed FCFS goodput WFQ goodput FCFS attainment WFQ attainment FCFS Jain WFQ Jain
11 1.75 req/s 2.15 req/s 28.5% 34.0% 0.673 0.937
23 2.31 req/s 1.89 req/s 37.5% 30.5% 0.720 0.875
37 2.33 req/s 2.15 req/s 39.5% 38.0% 0.769 0.895
53 2.78 req/s 2.63 req/s 49.0% 47.0% 0.718 0.824
71 2.28 req/s 2.49 req/s 40.5% 44.0% 0.842 0.990

Paired median effects (WFQ minus FCFS, 20,000 paired bootstrap resamples):

  • aggregate goodput: -0.15 req/s (95% interval -0.42 to +0.40) — no detectable aggregate cost or benefit;
  • all-offered attainment: -1.5 points (-7.0 to +5.5) — indistinguishable;
  • Jain fairness index over tenant attainment: +0.15 (+0.11 to +0.26), higher in every seed;
  • minimum tenant attainment: +11.1 points (+2.4 to +20.0); the minimum tenant was gold in every run;
  • gold attainment: +11.1 points (+2.4 to +20.0); batch attainment: -13.6 points (-24.2 to -7.7); standard: +1.1 points (-3.9 to +17.8).

Rejection became weight-aware: gold's rejection share fell from 27–46% under FCFS to 0–15% under WFQ, while batch's rose from 26–47% to 39–68%. The clean interpretation is that queue ordering does not change how much work the saturated engine completes; it changes which tenants' work meets its SLO, in proportion to declared weight.

One calibration finding matters for the next experiment: gold's declared 5,000 ms E2E SLO is infeasible for 37–66% of its own mixed-length requests even on an unloaded backend (long decodes alone exceed the SLO), which caps gold's all-offered attainment at 34–63% depending on the seed's trace. Restricted to feasible requests, WFQ raised gold attainment in every seed (e.g., 17.6% to 64.7% at seed 11); the weakest seed (23) still improved. scripts/analyze_policy_pairs.py reports these ceilings alongside the paired effects. Before M4's estimator-based admission, the tenant SLOs should either be recalibrated to the measured cost model or the estimator must treat infeasible requests as an explicit reject-at-admission class.

Raw artifacts are in results/raw/m3-policy-*/; the analysis is reproducible with scripts/analyze_policy_pairs.py results/raw --seeds 11 23 37 53 71. Strict-priority ordering and live paired runs at 60–120% of the measured knee remain open M3 items.