fix(test): lower gpu_memory_utilization for inference validation step (vllm-project#727)

## Summary

Fixes the DFlash OOM in nightly CI
([INFERENG-8705](https://redhat.atlassian.net/browse/INFERENG-8705)).

### Root cause

PR vllm-project#495 refactored `vllm_kwargs` handling in `test_offline_training.py`
to spread a single kwargs dict into both `launch_vllm_server_context`
(data-gen server) and `run_vllm_engine` (inference validation). This
inadvertently raised the inference step's `gpu_memory_utilization` from
**0.8** (the `run_vllm_engine` default) to **0.9**, leaving insufficient
headroom for vLLM's warmup allocation during speculative decoding.

The warmup allocation is **4.64 GiB** (`max_num_seqs=1024 ×
(num_spec_tokens+1)=8 × vocab_size=151,936 × 4 bytes`), introduced by
vLLM PR [#37812](vllm-project/vllm#37812) which
added spec-decode-aware warmup. With `gpu_memory_utilization=0.9` on an
80 GiB H100, only ~4.5 GiB remains free after model + KV cache — not
enough for the 4.64 GiB warmup logits tensor.

The failure is intermittent because it also depends on a ~1.83 GiB
background GPU process present on some CI runners but not others.

### Fix

Override `gpu_memory_utilization` back to **0.8** for the inference
validation step only. The data-gen server keeps 0.9 since it doesn't
perform spec decode warmup.

## Test plan

- [ ] CI nightly matrix passes (DFlash test specifically)
- [ ] Verify locally: `pytest tests/e2e/smoke/test_offline_training.py
-k dflash`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Orestis Zambounis <orestis.zambounis@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: orestis-z

tests/e2e/smoke/test_offline_training.py

@@ -165,6 +165,7 @@ def run_offline_e2e(
     # Step 4: Validate trained checkpoint with vLLM inference
     if prompts is not None:
         checkpoint_path = str(save_path / "checkpoint_best")
+        inference_kwargs = {**(vllm_kwargs or {}), "gpu_memory_utilization": 0.8}
         run_vllm_engine(
             model_path=checkpoint_path,
             tmp_path=tmp_path,
@@ -173,5 +174,5 @@ def run_offline_e2e(
             max_tokens=max_tokens,
             ignore_eos=ignore_eos,
             acceptance_thresholds=acceptance_thresholds,
-            **(vllm_kwargs or {}),
+            **inference_kwargs,
         )