Target release
Unsure
How were you running Hyperloom?
Local
Optimization domain
Inference
Issue type
Crash / failure
Which phase failed?
Optimization loop
Short Summary
GEAK uses GPU device 0 regardless of what was requested of the agent
Job ID (hosted UI runs only)
No response
Command / prompt submitted
Run a 8-hour Hyperloom xDiT optimization session for Tongyi-MAI/Z-Image-Turbo on the last available AMD GPU device. Use when the user wants a short xDiT diffusion-model demo (image generation, not LLM serving) on the local AMD ROCm environment.
Expected behavior
GEAK will leverage the same GPU asked of the agent when the optimization run is kicked off.
Actual behavior
GEAK used GPU device 0. In this particular case that GPU was already occupied so that lead to test failures.
Steps to reproduce
No response
Logs
Agent summary:
**`geak/handoff.json`'s `gpu_ids` field is hardcoded to `0` regardless of
the run's actual `ROCR_VISIBLE_DEVICES` pin, and GEAK's `bench_e2e.sh`
(`GPU=${GPU:-0}`) uses it to set `ROCR_VISIBLE_DEVICES=0` /
`HIP_VISIBLE_DEVICES=0` for every full server it launches** (baseline,
profile, and later validation/capture-selection legs) — confirmed via
`/proc/<pid>/environ` on the actually-launched `xdit` server subprocess,
even though the top-level `geak_runner.py`/`run_e2e.py` processes
correctly inherit the real pin. This silently collides with physical GPU
0 whenever something else on the host is using it. On a shared host, an
unrelated long-running `vllm-server` container occupying ~93-99% of GPU
0's VRAM caused this to reproduce 3 times in a single 2026-08-26 run
(before that container existed, the 2026-08-19/08-21 runs never hit
this, because GPU 0 was idle). The collision manifests as
`torch.OutOfMemoryError: HIP out of memory... GPU 0 has a total capacity
of 191.98 GiB of which 0 bytes is free` in
`geak/e2e_cycle0/baseline/replica_*/attempt_*/server.log` and
`.../profile/round_*/server.log`, and also shows up as GEAK's own
config-tuning specialist explicitly declining to A/B a candidate ("the
serving GPU (GPU 0) is held by a 186GB foreign co-tenant... must not be
touched") in its own workflow reasoning log. **The isolated per-op
opbench microbenchmark harness used for kernel-authoring bake-offs is
unaffected** — it runs in-process on the already-correctly-pinned GPU;
only GEAK's own full-server-launch paths (baseline/profile/config-tuning
validation) are degraded. Before trusting a `no_gain`/`incomplete` GEAK
baseline on a shared host, check `rocm-smi --showpids` for a foreign
tenant on physical GPU 0 and cross-check
`python3 -c "import json;print(json.load(open('geak/handoff.json'))['gpu_ids'])"`
against the run's actual `ROCR_VISIBLE_DEVICES` — a mismatch confirms
this bug rather than a real accuracy/perf regression. There is currently
no operator-facing override; it would need a Hyperloom-side fix to
forward the orchestrator's actual GPU pin into the handoff instead of
the literal `0` default.
Additional artifacts / links
No response
Data handling
Target release
Unsure
How were you running Hyperloom?
Local
Optimization domain
Inference
Issue type
Crash / failure
Which phase failed?
Optimization loop
Short Summary
GEAK uses GPU device 0 regardless of what was requested of the agent
Job ID (hosted UI runs only)
No response
Command / prompt submitted
Expected behavior
GEAK will leverage the same GPU asked of the agent when the optimization run is kicked off.
Actual behavior
GEAK used GPU device 0. In this particular case that GPU was already occupied so that lead to test failures.
Steps to reproduce
No response
Logs
Agent summary: **`geak/handoff.json`'s `gpu_ids` field is hardcoded to `0` regardless of the run's actual `ROCR_VISIBLE_DEVICES` pin, and GEAK's `bench_e2e.sh` (`GPU=${GPU:-0}`) uses it to set `ROCR_VISIBLE_DEVICES=0` / `HIP_VISIBLE_DEVICES=0` for every full server it launches** (baseline, profile, and later validation/capture-selection legs) — confirmed via `/proc/<pid>/environ` on the actually-launched `xdit` server subprocess, even though the top-level `geak_runner.py`/`run_e2e.py` processes correctly inherit the real pin. This silently collides with physical GPU 0 whenever something else on the host is using it. On a shared host, an unrelated long-running `vllm-server` container occupying ~93-99% of GPU 0's VRAM caused this to reproduce 3 times in a single 2026-08-26 run (before that container existed, the 2026-08-19/08-21 runs never hit this, because GPU 0 was idle). The collision manifests as `torch.OutOfMemoryError: HIP out of memory... GPU 0 has a total capacity of 191.98 GiB of which 0 bytes is free` in `geak/e2e_cycle0/baseline/replica_*/attempt_*/server.log` and `.../profile/round_*/server.log`, and also shows up as GEAK's own config-tuning specialist explicitly declining to A/B a candidate ("the serving GPU (GPU 0) is held by a 186GB foreign co-tenant... must not be touched") in its own workflow reasoning log. **The isolated per-op opbench microbenchmark harness used for kernel-authoring bake-offs is unaffected** — it runs in-process on the already-correctly-pinned GPU; only GEAK's own full-server-launch paths (baseline/profile/config-tuning validation) are degraded. Before trusting a `no_gain`/`incomplete` GEAK baseline on a shared host, check `rocm-smi --showpids` for a foreign tenant on physical GPU 0 and cross-check `python3 -c "import json;print(json.load(open('geak/handoff.json'))['gpu_ids'])"` against the run's actual `ROCR_VISIBLE_DEVICES` — a mismatch confirms this bug rather than a real accuracy/perf regression. There is currently no operator-facing override; it would need a Hyperloom-side fix to forward the orchestrator's actual GPU pin into the handoff instead of the literal `0` default.Additional artifacts / links
No response
Data handling