fish-speech-s2-pro on localai/localai:latest-nvidia-l4t-arm64-cuda-13 (DGX Spark) #11344

Summary

On arm64 + CUDA 13 (NVIDIA DGX Spark / GB10), the fish-speech backend is unusable out of the box. Four distinct defects stack on top of each other; each has to be cleared before the next one surfaces.

All four were reproduced on real hardware and fixed with runtime patches; the backend now loads on GPU and serves voice-cloned TTS correctly.

Environment

• Image: localai/localai:latest-nvidia-l4t-arm64-cuda-13
• Backend: cuda13-nvidia-l4t-arm64-fish-speech
• Model: fish-speech-s2-pro (gallery)
• Hardware: NVIDIA DGX Spark — GB10, compute capability 12.1 (sm_121), aarch64
• Driver 580.95.05, CUDA 13.0. nvidia-smi works inside the container, so GPU passthrough is not the issue
• Backend venv Python 3.10.18
• Deployed with Docker via Coolify

uname -a: Linux 670a7ffeb200 6.14.0-1015-nvidia #15-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 25 18:02:16 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

Defect 1 — PYTHONPATH is never set

Symptom

Initializing libbackend for cuda13-nvidia-l4t-arm64-fish-speech
Using portable Python
Added /backends/cuda13-nvidia-l4t-arm64-fish-speech/lib to LD_LIBRARY_PATH for GPU libraries
Server started. Listening on: 127.0.0.1:41285
CUDA is not available
Downloading model from HuggingFace: fishaudio/s2-pro
Fetching 13 files: 100%|██████████| 13/13
Using device: cpu, precision: torch.float32, compile: False
[ERROR] Loading model: ModuleNotFoundError: No module named 'fish_speech.inference_engine'
Traceback (most recent call last):
  File "/backends/cuda13-nvidia-l4t-arm64-fish-speech/backend.py", line 205, in LoadModel
    from fish_speech.inference_engine import TTSInferenceEngine
ModuleNotFoundError: No module named 'fish_speech.inference_engine'
Received termination signal. Shutting down...

Root cause

install.sh clones the upstream tree to ${EDIR}/fish-speech-src and installs it editable, with this comment:

# Install fish-speech deps from source (without the package itself since we use PYTHONPATH)
pip install ${EXTRA_PIP_INSTALL_FLAGS:-} -e "${FISH_SPEECH_DIR}"

But nothing ever sets PYTHONPATH. grep -n PYTHONPATH backend/python/common/libbackend.sh returns zero matches, and run.sh only sources the library and calls startBackend.

The editable-install finder records the build-time absolute path, which does not survive relocation into /backends/<backend>/ — the same relocation hazard libbackend.sh already documents for venv paths:

# $EDIR (resolved at runtime via realpath) instead of the path baked into
# bin/activate at venv-create time. `uv venv` (and `python -m venv`) both bake
# the create-time absolute path in, so sourcing activate on a relocated venv
# ... silently prepends a stale, non-existent path to $PATH.

The venv path rewrite handles bin/activate, but not the editable finder. The parent package still resolves as an empty namespace, so only the submodule import fails — which is why the error names fish_speech.inference_engine rather than fish_speech.

Proposed fix — backend/python/fish-speech/run.sh

 fi
 
+# install.sh clones the upstream tree to ${EDIR}/fish-speech-src and installs it
+# editable. The editable-install finder records the build-time absolute path,
+# which does not survive relocation to /backends/<backend>/, so expose the
+# source tree explicitly here. libbackend.sh calls init() at the bottom of the
+# file, so EDIR is already resolved at this point.
+export PYTHONPATH="${EDIR}/fish-speech-src${PYTHONPATH:+:${PYTHONPATH}}"
+
 startBackend $@

The ${VAR:+:${VAR}} form is deliberate — a trailing colon would put the CWD on sys.path.

Defect 2 — torch resolves to the CPU-only wheel on aarch64

Symptom

$ /backends/cuda13-nvidia-l4t-arm64-fish-speech/venv/bin/python -c \
    "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
2.8.0+cpu None False

An 11 GB model silently loads onto the CPU: Using device: cpu, precision: torch.float32.

Root cause

requirements-l4t13.txt:

--extra-index-url https://download.pytorch.org/whl/cu130
torch
torchaudio

The cu130 index publishes no torch 2.8.0 wheel for either architecture — only 2.9.0, 2.9.1, 2.12.1 and 2.13.0:

$ curl -s https://download.pytorch.org/whl/cu130/torch/ | grep cp310 | grep aarch64
torch-2.9.0+cu130-cp310-cp310-manylinux_2_28_aarch64.whl
torch-2.9.1+cu130-cp310-cp310-manylinux_2_28_aarch64.whl
torch-2.12.1+cu130-cp310-cp310-manylinux_2_28_aarch64.whl
torch-2.13.0+cu130-cp310-cp310-manylinux_2_28_aarch64.whl

--extra-index-url does not enforce priority, so pip falls back to PyPI. On x86_64 the PyPI default torch is a CUDA build and this stays invisible; on aarch64 it is CPU-only. That asymmetry is why only the arm64 image is affected.

requirements-cublas13.txt is byte-identical and carries the same latent problem.

Proposed fix — requirements-l4t13.txt and requirements-cublas13.txt

 --extra-index-url https://download.pytorch.org/whl/cu130
-torch
-torchaudio
+# The cu130 index has no 2.8.x wheels for aarch64 or x86_64. Without a pin,
+# --extra-index-url lets pip fall back to PyPI, which on aarch64 serves a
+# CPU-only torch and silently disables GPU inference. The +cu130 local version
+# does not exist on PyPI, so it forces resolution from the PyTorch index.
+torch==2.9.1+cu130
+torchaudio==2.9.1

Defect 3 — incomplete cuDNN bundle in the backend's lib/

Symptom

RuntimeError('CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed
ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH')

Root cause

The image ships a partial cuDNN 9.24.0 family — the libcudnn.so.9 dispatcher plus only 3 of the 7 sublibraries listed in CUDNN9_SUBLIBS:

Missing: libcudnn_adv, libcudnn_engines_precompiled, libcudnn_engines_runtime_compiled, libcudnn_heuristic.

libbackend.sh prepends the backend's own lib dir:

if [ -d "${EDIR}/lib" ]; then
    export LD_LIBRARY_PATH="${EDIR}/lib:${LD_LIBRARY_PATH:-}"
fi

So the dispatcher resolves to 9.24.0 while the four absent sublibraries fall through to the torch wheel's 9.13.0. Because the prepend happens inside startBackend, nothing exported from run.sh can override it — the stale libraries have to be physically removed.

Note: defects 2 and 3 are causally linked

Dockerfile.python bundles cuDNN only when the venv has no pip cuDNN:

# cuDNN from pip, and bundles one only when it does not (issue #10905).
RUN mkdir -p /${BACKEND}/lib && \
    TARGET_LIB_DIR="/${BACKEND}/lib" ... bash /package-gpu-libs.sh "/${BACKEND}/lib"

Because the CPU torch wheel from defect 2 brings no cuDNN, the bundling path was taken — and it emitted a partial set despite scripts/build/package-gpu-libs.sh documenting the hazard explicitly:

# The cuDNN 9 sublibraries that must always travel together. The dispatcher
# libcudnn.so.9 is a thin shim that dlopen()s these by bare soname on first use,
# ... See verify_cudnn_bundle for why a partial set is fatal.
CUDNN9_SUBLIBS=(
    libcudnn_adv
    libcudnn_cnn
    libcudnn_engines_precompiled
    libcudnn_engines_runtime_compiled
    libcudnn_graph
    libcudnn_heuristic
    libcudnn_ops
)

verify_cudnn_bundle apparently did not fail the build. Fixing defect 2 may make this moot for fish-speech (pip cuDNN present → no bundling), but the verification gap looks worth closing on its own.

Defect 4 — torchaudio ≥ 2.9 requires torchcodec, which has no linux-aarch64 wheels

Symptom

Voice cloning with a reference audio file:

[INFO] Using per-request reference audio: /data/voice-profiles/.leases/.../....wav
Generating speech for text: ...
Error in TTS: TorchCodec is required for load_with_torchcodec. Please install torchcodec to use this function.
Traceback (most recent call last):
  File ".../torchaudio/_torchcodec.py", line 82, in load_with_torchcodec
    from torchcodec.decoders import AudioDecoder
ModuleNotFoundError: No module named 'torchcodec'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/backends/cuda13-nvidia-l4t-arm64-fish-speech/backend.py", line 396, in TTS
    for result in self.engine.inference(tts_request)
  File ".../fish-speech-src/fish_speech/inference_engine/__init__.py", line 55, in inference
    prompt_tokens, prompt_texts = self.load_by_hash(
  File ".../fish-speech-src/fish_speech/inference_engine/reference_loader.py", line 113, in load_by_hash
    self.encode_reference(
  File ".../fish-speech-src/fish_speech/inference_engine/vq_manager.py", line 31, in encode_reference
    reference_audio_content = self.load_audio(reference_audio, sample_rate)
  File ".../fish-speech-src/fish_speech/inference_engine/reference_loader.py", line 141, in load_audio
    waveform, original_sr = torchaudio.load(reference_audio, backend=self.backend)
  File ".../torchaudio/__init__.py", line 86, in load
    return load_with_torchcodec(
ImportError: TorchCodec is required for load_with_torchcodec.

Root cause

torchaudio 2.9 dropped its own I/O backends and routes torchaudio.load() through torchcodec. fish-speech partially accounts for 2.9 (# list_audio_backends() was removed in torchaudio 2.9) and still passes backend=self.backend, but 2.9's load() ignores it and goes to load_with_torchcodec regardless.

torchcodec publishes no linux-aarch64 wheels at all. Checked every release from 0.7.0 through 0.9.1 on PyPI — only macosx_11_0_arm64, manylinux_2_28_x86_64 and win_amd64. Installing it on arm64 would require a source build with FFmpeg dev headers and cmake, which is not viable inside this backend image.

So on arm64, any torchaudio ≥ 2.9 makes voice cloning impossible. Note this is not hypothetical for LocalAI: the fix for defect 2 pins torchaudio==2.9.1, which lands exactly in this state.

Proposed fix — backend/python/fish-speech/install.sh

soundfile is already a backend requirement, and install.sh already patches the cloned tree (the existing pyaudio removal), so this follows established practice in the same file:

 # Remove pyaudio from fish-speech deps ...
 sed -i.bak '/"pyaudio"/d' "${FISH_SPEECH_DIR}/pyproject.toml"
 
+# torchaudio >= 2.9 routes torchaudio.load() through torchcodec, which publishes
+# no linux-aarch64 wheels (checked 0.7.0-0.9.1 on PyPI: macOS arm64, linux
+# x86_64 and win_amd64 only). Voice cloning would be impossible on arm64.
+# soundfile is already in requirements.txt and reads both paths and BytesIO,
+# so use it for the reference-audio load instead.
+sed -i 's|waveform, original_sr = torchaudio.load(reference_audio, backend=self.backend)|import soundfile as _sf\n        _d, original_sr = _sf.read(reference_audio, dtype="float32", always_2d=True)\n        waveform = torch.from_numpy(_d.T.copy())|' \
+    "${FISH_SPEECH_DIR}/fish_speech/inference_engine/reference_loader.py"
+

always_2d=True yields [N, C]; .T gives the [C, N] layout the channel-averaging code immediately below already expects.

The cleaner long-term fix belongs upstream in fish-speech (pin torchaudio<2.9, or drop to soundfile there). Happy to open that PR too if preferred.

Scope check

The other torchaudio.load call sites in the cloned tree are not on the gRPC serving path:

Output is unaffected: LocalAI's backend.py writes with soundfile (sf.write(request.dst, audio_data, sample_rate), line 422), not torchaudio.save, so it never reaches save_with_torchcodec.

Reproduction

1. Run localai/localai:latest-nvidia-l4t-arm64-cuda-13 on an aarch64 CUDA 13 host with GPU passthrough
2. local-ai backends install fish-speech
3. Load the gallery model fish-speech-s2-pro → HTTP 500, backend exits with ModuleNotFoundError (defect 1)
4. After working around 1: backend loads but logs CUDA is not available and falls back to CPU (defect 2)
5. After working around 2: loads on GPU, first TTS request fails with the cuDNN mismatch (defect 3)
6. After working around 3: plain TTS works; TTS with reference audio fails with the torchcodec ImportError (defect 4)

Expected behavior

fish-speech-s2-pro loads on GPU and serves TTS — including voice cloning with reference audio — on arm64 + CUDA 13, without manual patching of the backend image.

Verification

Runtime patches applied inside the running container. With all four, the backend loads as Using device: cuda, precision: torch.bfloat16 and voice-cloned TTS completes successfully.

BD=/backends/cuda13-nvidia-l4t-arm64-fish-speech

# 1 — expose the cloned source tree
git clone --depth 1 https://github.com/fishaudio/fish-speech.git "$BD/fish-speech-src"
sed -i 's|^startBackend|export PYTHONPATH="'"$BD"'/fish-speech-src${PYTHONPATH:+:${PYTHONPATH}}"\nstartBackend|' "$BD/run.sh"

# 2 — replace the CPU wheel (venv has no pip; LocalAI builds it with uv)
"$BD/venv/bin/python" -m ensurepip --upgrade
"$BD/venv/bin/python" -m pip install --extra-index-url https://download.pytorch.org/whl/cu130 \
  "torch==2.9.1+cu130" "torchaudio==2.9.1"

# 3 — drop the partial cuDNN so the complete wheel family is used
mkdir -p "$BD/lib.disabled" && mv "$BD/lib"/libcudnn* "$BD/lib.disabled"/

# 4 — bypass torchcodec
sed -i 's|^        waveform, original_sr = torchaudio.load(reference_audio, backend=self.backend)$|        import soundfile as _sf\n        _d, original_sr = _sf.read(reference_audio, dtype="float32", always_2d=True)\n        waveform = torch.from_numpy(_d.T.copy())|' \
  "$BD/fish-speech-src/fish_speech/inference_engine/reference_loader.py"

Post-fix checks:

$ "$BD/venv/bin/python" -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
2.9.1+cu130 13.0 True

$ "$BD/venv/bin/python" -c "import torch; print(torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))"
NVIDIA GB10 (12, 1)

Note that a full container restart is required after patching — LocalAI keeps the backend as a separate long-lived process, so reloading the model alone leaves the pre-patch module object in memory.

Caveat

Everything above was verified by patching the running container. I have not rebuilt the arm64 CUDA 13 backend image, so the proposed source changes are unverified at image-build time. Happy to test any build on real GB10 hardware.