[noahark]

Problem and result

Measured data for the fully-resident big-VRAM regime, as a complement to the 2026-11-02 write-up
(bench/results/2026-11-02-4gpu-4x128k), whose 16 GB cards tiered experts to RAM. Same two questions on the
opposite memory regime - 3x V100 32 GB, everything resident, no RAM tier, no small card in the split:

• Plain --parallel 4 is the sweet spot: 98.0 tok/s in all (4 x 24.5), 1.61x solo. 8 streams fall
  back to 90.6.
• The rows family halves it here too: MAYA_GLM_SPLIT_ROWS=4 on --parallel 4 gives 51.0
  (4 x 12.8). This agrees with the 2026-11-02 write-up's "tried and slower" table (SPLIT_ROWS=4:
  8.3-8.6 vs 20-21 there) - on a fully-resident shape the rows engine has no PCIe streaming to avoid and
  pays the row serialization instead. ROWS_ATTN / MULTI_MTP / PREFILL_INTERLEAVE build on
  SPLIT_ROWS and were not measured further for that reason.
• Prefill does not scale with streams (not covered by the earlier write-up): four concurrent 12.1-12.2k
  prompts each prefill at full solo speed (1040-1045 tok/s vs 1049 solo), but serialized - three of four
  wait ~37 s, every request's wall is 51 s, aggregate ~1043 tok/s. The payoff of --parallel on this shape
  is decode-only.

Related discussion

Extends bench/results/2026-11-02-4gpu-4x128k (the 4-conversations-at-128K write-up) with the fully-resident
regime and the prefill-vs-streams observation. Data only - no code change.

Approach

Results README under bench/results/2026-11-14-3xv100-32gb-glm-concurrency/, in the folder's usual format:
rig, run config, the solo bands, the concurrency table (solo / plain 4 / plain 8 / SPLIT_ROWS=4), the
prefill-vs-streams table, and the client description. Greedy client (temperature 0) for the concurrency
table, max_tokens 2048 with a ~15-token "Count from 1 to 1200" prompt so the whole window is decode;
prefill measured separately with 3400-pseudo-word prompts and max_tokens 48.

Validation

Passed:

• Rig: 3x Tesla V100-PCIE-32GB (TCC) + 2x Xeon E5-2696 v3, 128 GB DDR3L-1600 ECC REG, Windows 10 Enterprise
  LTSC 2021; weights on NVMe. Engine v1.1.63 (source build, sm_70, CUDA 12.4), Maya-S-v2-IQ2_XXS, layer
  split (auto) [0,17)/[17,32)/[32,45], served through serve/server.py --engine maya --port 8080.
• Solo bands, 3 rounds each, server-side timings from the engine's own glm prefill: / glm stat: lines:
  short decode 59.5-63.3, long prefill 1045-1053, long decode 46.6, vram hit 99.5%, drafts accepted 80-87%.
• Concurrency arms: plain 4 (98.0 aggregate, every request at its 2048 cap), plain 8 (90.6),
  SPLIT_ROWS=4 (51.0) - each a fresh engine start, warm-up rounds not counted, greedy output.
• Prefill arms: 1 stream (1045-1053 tok/s, 3 rounds) and 4 streams (1040-1045 each, walls 51 s) - the
  engine's glm prefill: per-card chunk lines show one request's chunks owning the pipeline at a time.

Failed:

• none observed.

Not run:

• ROWS_ATTN / MULTI_MTP / PREFILL_INTERLEAVE (they require SPLIT_ROWS, which is -48% here before
  their gains could apply). Linux. HIP/SYCL/Metal builds (data only, no code change).

Limits

One rig, one model size (Maya-S-v2-IQ2_XXS). The rows-family result covers SPLIT_ROWS=4 with 4
conversations; 8 rows with 8 conversations was not measured on this rig.