fix(rocm): unset empty HSA_OVERRIDE_GFX_VERSION before torch loads

[Xarianne]

Problem

When running Voicebox with ROCm in Docker, no GPU is detected even when the hardware is present and ROCm PyTorch is installed:

RuntimeError: No CUDA GPUs are available

Root cause

docker-compose.rocm.yml sets:

HSA_OVERRIDE_GFX_VERSION=${HSA_OVERRIDE_GFX_VERSION:-}

This uses the shell default syntax, which sets the variable to an empty string when no value is provided. Docker compose cannot conditionally omit an environment variable — it either sets it (even to empty) or doesn't include the line at all.

An empty string is not the same as unset. ROCm's HSA runtime interprets HSA_OVERRIDE_GFX_VERSION="" as "force the GPU version to empty", which causes it to find zero GPUs. This affects all GPUs, including ones that are natively supported by ROCm (e.g. gfx1201 / RX 9070 on ROCm 7.2) and don't need an override at all.

Verified inside the container:

• Empty string → torch.cuda.is_available() = False, 0 devices
• Unset → True, 2 devices
• 12.0.0 → True, 2 devices

Fix

In backend/app.py, before torch is imported, pop the environment variable when it is empty:

if not os.environ.get("HSA_OVERRIDE_GFX_VERSION"):
    os.environ.pop("HSA_OVERRIDE_GFX_VERSION", None)

This ensures that when docker-compose passes an empty string, the variable is fully removed from the environment before ROCm initialises. Users who explicitly set HSA_OVERRIDE_GFX_VERSION for older GPUs (RDNA 1-3) are not affected — the variable is only popped when it is empty.

Testing

Tested on RX 9070 (gfx1201) with ROCm 7.2 and PyTorch 2.12.1+rocm7.2:

• Without fix: torch.cuda.is_available() = False, 0 devices
• With fix: torch.cuda.is_available() = True, 2 devices (AMD Radeon RX 9070 + AMD Ryzen 7 9800X3D iGPU)

Summary by CodeRabbit

• Bug Fixes
  • Improved GPU startup behavior by ignoring empty HSA_OVERRIDE_GFX_VERSION values before hardware detection runs.
  • Prevents an unset override from being treated as a forced empty GPU configuration, helping avoid ROCm runtime issues.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3f9741f5-7e34-4542-8645-b9451b48a036

📥 Commits

Reviewing files that changed from the base of the PR and between f2cf2a7 and ec02f65.

📒 Files selected for processing (1)

• backend/app.py

---

📝 Walkthrough

Walkthrough

Adds a guard in backend/app.py that removes HSA_OVERRIDE_GFX_VERSION from the process environment when it is unset or empty, executed before existing ROCm/GPU detection and override logic.

Changes

HSA Override Env Var Fix

Layer / File(s)	Summary
Guard against empty HSA_OVERRIDE_GFX_VERSION backend/app.py	Pops the HSA_OVERRIDE_GFX_VERSION environment variable when missing or falsy, before ROCm/GPU detection runs, to prevent an empty value being treated as a forced override.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

• jamiepine/voicebox#630: Sets HSA_OVERRIDE_GFX_VERSION in docker-compose.rocm.yml, directly complementing the environment-clearing logic added here.
• jamiepine/voicebox#785: Modifies the same backend/app.py startup logic around conditionally setting HSA_OVERRIDE_GFX_VERSION based on rocminfo detection.
• jamiepine/voicebox#817: Makes HSA_OVERRIDE_GFX_VERSION optional, addressing the same environment variable handling point.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main fix: unsetting an empty HSA_OVERRIDE_GFX_VERSION before PyTorch loads.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}