[CI/Build] Request one GPU per replica in the two-pods Helm functionality test

[Shaoting-Feng]

Summary

Two-Pods-Minimal-Example in the Helm functionality workflow fails intermittently: one of the two vLLM replicas exits at ~80–90 s, restarts, and the validation step then overruns its 3-minute timeout a few seconds after printing Requests were successful. (e.g. run 32555187470, run 32516740201). Root cause: the two replicas end up sharing one GPU and vLLM's startup memory profiling sees each other's allocations.

Why

• tests/assets/values-01-2pods-minimal-example.yaml sets requestGPU: 0.5. The chart only renders a GPU request/limit when int(requestGPU) > 0 (chart.resources / chart.hasLimits in helm/templates/_helpers.tpl), so 0.5 silently becomes no GPU request at all. Both pods see every GPU on the runner and both pick cuda:0; the runner's second L4 is never used.

• Both engines finish loading at the same moment and run vLLM's memory profiling concurrently. vLLM measures device-wide free memory, so whichever replica allocates its ~7.7 GiB KV cache first gets charged against the sibling's --gpu-memory-utilization 0.4 budget. Reproduced on the CI runner with the unmodified values file — 10 trials, 5/10 showed the squeeze, e.g. same second, same GPU:

  • pod A: Available KV cache memory: 7.69 GiB
  • pod B: Available KV cache memory: 0.24 GiB … Replace gpu_memory_utilization config with --kv-cache-memory=-263896780 (-0.25 GiB)

  Two trials went down to 0.06 GiB (a 1,712-token KV cache) — roughly 25 MiB from the point where vLLM can no longer fit one max_model_len=1024 request, raises ValueError, and the engine exits → container restart → the CI signature above. The "squeezed but alive" cases pass CI on a crippled engine.

• It is not a real OOM (GPU 0 peaks at ~18 of 23 GB) and not the shared HF cache (no errors in any engine log). Healthy single-engine startup is stable at ~90 s, so the 3-minute step budget is fine once the restart goes away.

Changes

• requestGPU: 0.5 → 1 so each replica gets its own nvidia.com/gpu and the two engines no longer profile against each other. The runner has two L4s, and values-04-multiple-models.yaml in the same workflow already schedules two engines with requestGPU: 1 each. --gpu-memory-utilization 0.4 is left as is.
• Deliberately not raising the 3-minute timeout (that only hides the restart), and not touching tutorials/assets/values-01-2pods-minimal-example.yaml (same 0.5, but user-facing). The chart silently truncating fractional requestGPU to "no GPU request" deserves its own follow-up (template guard and/or docs).

Testing

• Two-Pods-Minimal-Example in this PR's CI run exercises the changed file directly.
• By inspection of _helpers.tpl, integer requestGPU renders nvidia.com/gpu: "1" under both requests and limits.

Related: #1030, #1036.

---

• Make sure the code changes pass the pre-commit checks.
• Sign-off your commit by using -s when doing git commit
• Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

[gemini-code-assist]

Code Review

This pull request updates the minimal example configuration file tests/assets/values-01-2pods-minimal-example.yaml to increase the requested GPU resources from 0.5 to 1. There are no review comments, and I have no feedback to provide.

[ruizhang0101]

LGTM