fix(rocm): add MIOpen stability env vars to docker-compose.rocm.yml

[Xarianne]

Problem

When running Voicebox with ROCm on RDNA4 GPUs (e.g. RX 9070), inference fails with miopenStatusUnknownError and causes severe system stuttering with repeated MIOpen warnings:

MIOpen(HIP): Warning [IsEnoughWorkspace] [EvaluateInvokers] Solver <GemmFwdRest>, workspace required: 516096000, provided ptr: 0 size: 0

Two issues:

1. Without explicit cache paths, MIOpen fails to write its kernel cache in a fresh container, leading to miopenStatusUnknownError.
2. MIOpen's default exhaustive benchmarking mode tries every kernel variant on every generation. On RDNA4, many kernels fail to allocate workspace memory (returning ptr: 0 size: 0), causing MIOpen to thrash through failing kernels and stall the GPU.

Fix

Add three environment variables to docker-compose.rocm.yml:

Variable	Purpose
MIOPEN_USER_DB_PATH	Redirect MIOpen kernel cache to a writable, persistent directory
MIOPEN_CUSTOM_CACHE_DIR	Same, for custom operator cache
MIOPEN_FIND_MODE=FAST	Use heuristic kernel selection instead of exhaustive benchmarking

MIOPEN_FIND_MODE=FAST does not affect output quality. All MIOpen kernel variants produce the same numerical result; fast mode simply selects a known-good kernel using heuristics instead of benchmarking every variant on the GPU.

Testing

Tested on RX 9070 (gfx1201) with ROCm 7.2 and PyTorch 2.12.1+rocm7.2. Each variable was tested in isolation:

• Without cache path vars: miopenStatusUnknownError on inference
• Without MIOPEN_FIND_MODE=FAST: system stuttering during inference, repeated MIOpen workspace warnings on every generation even when cache is present
• With all three variables: clean logs, no warnings, smooth inference, voice generation works correctly

Hardware note

This was tested on a Ryzen 7 9800X3D + RX 9070 with a Gigabyte B650M DS3H motherboard.

Dependencies

This PR depends on several other fixes that must be in place before the MIOpen fixes can be tested or used:

• Update .dockerignore #861 — exempts scripts/rocm-entrypoint.sh from .dockerignore. Without this, the ROCm Docker build is broken because the entrypoint script is excluded from the build context.
• Fix HF cache permissions and add startup fallback #677 — fixes HuggingFace cache permissions so the container can write model files. The MIOpen cache paths point to /app/data/cache/, which requires the volume permissions to be correct. See my suggestion on that issue with alternative approach.
• Fix/docker permision #584 — fixes Docker permission issues with the voicebox user UID and bind-mounted directories. Without correct permissions, MIOpen cannot write its cache to the data volume. See my suggestion on that issue with alternative approach.
• fix(rocm): unset empty HSA_OVERRIDE_GFX_VERSION before torch loads #864 HSA override fix: unsets the empty HSA_OVERRIDE_GFX_VERSION environment variable that docker-compose passes by default. Without this, no GPU is detected at all.

Summary by CodeRabbit

• New Features
  • Improved ROCm/AMD GPU startup behavior by preserving MIOpen cache and database data in a persistent location.
  • Enabled faster kernel selection to reduce initialization time for supported GPU workloads.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds three MIOpen-related environment variables to the ROCm Docker Compose override: redirecting the MIOpen user database path and custom cache directory to persistent storage under /app/data/cache, and setting MIOPEN_FIND_MODE=FAST for faster kernel selection.

Changes

ROCm MIOpen Environment Configuration

Layer / File(s)	Summary
MIOpen cache and find mode settings docker-compose.rocm.yml	Adds MIOPEN_USER_DB_PATH, MIOPEN_CUSTOM_CACHE_DIR pointing to persistent cache paths, and MIOPEN_FIND_MODE=FAST to skip exhaustive kernel benchmarking.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

• jamiepine/voicebox#630: Extends the same docker-compose.rocm.yml overlay with related MIOpen kernel/cache environment settings.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main ROCm/MIOpen environment-variable change in docker-compose.rocm.yml.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Xarianne]

This PR is the result of extensive testing on an RDNA4 card (RX 9070). As of commit f2cf2a7 Voicebox does not work on Linux with AMD GPUs via Docker. The above changes, combined with the dependency PRs, result in a working Voicebox: fixing build failures, model download failures, and inference failures.

I went through existing PRs, made suggestions to existing ones, and added new ones where needed. I can confirm that applying all of the above results in at least a minimum viable product where an RDNA4 card (at least the 9070) can run without issues.

[Xarianne]

I have just filed an issue upstream about this btw: ROCm/MIOpen#3957

It looks like they had this issue before with RDNA 3 cards, they seem to close the issues but the problem reappears regularly.