Skip to content

GEAK uses first GPU device regardless of top-level agent selection for xDiT run #1312

Description

@shaunpamd

Target release

Unsure

How were you running Hyperloom?

Local

Optimization domain

Inference

Issue type

Crash / failure

Which phase failed?

Optimization loop

Short Summary

GEAK uses GPU device 0 regardless of what was requested of the agent

Job ID (hosted UI runs only)

No response

Command / prompt submitted

Run a 8-hour Hyperloom xDiT optimization session for Tongyi-MAI/Z-Image-Turbo on the last available AMD GPU device. Use when the user wants a short xDiT diffusion-model demo (image generation, not LLM serving) on the local AMD ROCm environment.

Expected behavior

GEAK will leverage the same GPU asked of the agent when the optimization run is kicked off.

Actual behavior

GEAK used GPU device 0. In this particular case that GPU was already occupied so that lead to test failures.

Steps to reproduce

No response

Logs

Agent summary:

    **`geak/handoff.json`'s `gpu_ids` field is hardcoded to `0` regardless of
    the run's actual `ROCR_VISIBLE_DEVICES` pin, and GEAK's `bench_e2e.sh`
    (`GPU=${GPU:-0}`) uses it to set `ROCR_VISIBLE_DEVICES=0` /
    `HIP_VISIBLE_DEVICES=0` for every full server it launches** (baseline,
    profile, and later validation/capture-selection legs) — confirmed via
    `/proc/<pid>/environ` on the actually-launched `xdit` server subprocess,
    even though the top-level `geak_runner.py`/`run_e2e.py` processes
    correctly inherit the real pin. This silently collides with physical GPU
    0 whenever something else on the host is using it. On a shared host, an
    unrelated long-running `vllm-server` container occupying ~93-99% of GPU
    0's VRAM caused this to reproduce 3 times in a single 2026-08-26 run
    (before that container existed, the 2026-08-19/08-21 runs never hit
    this, because GPU 0 was idle). The collision manifests as
    `torch.OutOfMemoryError: HIP out of memory... GPU 0 has a total capacity
    of 191.98 GiB of which 0 bytes is free` in
    `geak/e2e_cycle0/baseline/replica_*/attempt_*/server.log` and
    `.../profile/round_*/server.log`, and also shows up as GEAK's own
    config-tuning specialist explicitly declining to A/B a candidate ("the
    serving GPU (GPU 0) is held by a 186GB foreign co-tenant... must not be
    touched") in its own workflow reasoning log. **The isolated per-op
    opbench microbenchmark harness used for kernel-authoring bake-offs is
    unaffected** — it runs in-process on the already-correctly-pinned GPU;
    only GEAK's own full-server-launch paths (baseline/profile/config-tuning
    validation) are degraded. Before trusting a `no_gain`/`incomplete` GEAK
    baseline on a shared host, check `rocm-smi --showpids` for a foreign
    tenant on physical GPU 0 and cross-check
    `python3 -c "import json;print(json.load(open('geak/handoff.json'))['gpu_ids'])"`
    against the run's actual `ROCR_VISIBLE_DEVICES` — a mismatch confirms
    this bug rather than a real accuracy/perf regression. There is currently
    no operator-facing override; it would need a Hyperloom-side fix to
    forward the orchestrator's actual GPU pin into the handoff instead of
    the literal `0` default.

Additional artifacts / links

No response

Data handling

  • I confirmed the attached logs/artifacts contain no sensitive information

Metadata

Metadata

Assignees

No one assigned

    Labels

    domain:inferenceRelated to inference optimizationtype:bugSomething is broken or not working as expected

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions