Fix Canary-Qwen CUDA-tag skew + cdifflib header build

[wilke0818]

Summary

The Canary-Qwen subprocess venv currently fails to come up on a clean install for two unrelated reasons. This PR fixes both with the minimum-footprint changes that don't
disturb other subprocess venvs (Sortformer, Conformer-CTC, Coqui, PPGs, etc.) or alter the resolver behavior for environments where things already work.

Problem 1: torchaudio CUDA-tag skew

PyPI ships torch 2.8.x as +cu129 but torchaudio 2.8.x as +cu128. A fresh ensure_venv install of the Canary-Qwen requirements therefore produces a venv where torch is linked
against CUDA 12.9 and torchaudio against CUDA 12.8. The first torchaudio import then raises:

RuntimeError: Detected that PyTorch and TorchAudio were compiled with
different CUDA versions. PyTorch has CUDA version 12.9 whereas TorchAudio
has CUDA version 12.8.

Fix. After ensure_venv returns, run a one-shot uv pip install --force-reinstall --no-deps --index-url https://download.pytorch.org/whl/cu128 torch>=2.8,<2.9
torchaudio>=2.8,<2.9. The PyTorch cu128 wheel index always publishes torch + torchaudio together with matching +cu128 tags, so the two wheels are guaranteed to align. A
marker file (.senselab-canary-cuda-fixup) inside the venv directory gates this to first-build only — subsequent calls reuse the venv as-is with zero overhead.

Why this doesn't break other systems

• Scope is limited to Canary-Qwen. The fixup block lives in canary_qwen.py, not in subprocess_venv.py. No other backend's venv is touched. Sortformer, Conformer-CTC, Coqui,
  PPGs, etc. continue resolving exactly as before.
• --no-deps means only the two affected wheels are swapped. The NeMo install, the nvidia-*-cu12 CUDA runtime packages installed during the main resolver pass, and all other
  transitive dependencies stay in place. cu128 torch links against libcudart.so.12 (provided by the nvidia-cuda-runtime-cu12 already present from the main install).
• cu128 runtime is broadly compatible. CUDA drivers are forward-compatible within and across the next major version: a host running cu12.6, cu12.8, cu12.9, or cu13.x can all
  execute code compiled for cu12.8. The fix works on any of those host drivers — no host-CUDA mirroring required.
• --force-reinstall is bounded by the marker. It runs exactly once per venv lifetime. After the marker is written, the block is a no-op (a single Path.is_file() check).
• Bandwidth cost is one-time and small (~600 MB for two wheels), and zero on every subsequent call. The full 5 GB venv is not rebuilt.

Problem 2: cdifflib source build fails on HPC hosts

cdifflib is a transitive dep of NeMo that lacks pre-built wheels for some Python/platform pairs and falls back to a C source build. Spack-built Pythons on HPC clusters (e.g.,
MIT ORCD) are typically installed without development headers, so the build fails with fatal error: Python.h: No such file or directory.

Fix. Pass --managed-python to uv venv when creating subprocess venvs. uv's managed Python downloads ship standalone Python builds that include development headers, so C
extensions compile cleanly.

Why this doesn't break other systems

• --managed-python is opt-in for uv, not a mode switch. It tells uv to use its own downloaded Python interpreter instead of a system one. On machines where uv would already
  pick a managed Python (the common case when the requested version isn't on PATH), the flag is a no-op. On machines where the system Python does have headers (most laptops,
  most CI runners), the managed Python is functionally equivalent.
• All subprocess venvs benefit uniformly. The flag is set in the shared ensure_venv helper, so any backend with a build-from-source transitive dep gets the same fix.
• No additional disk cost in typical cases. uv caches its managed Pythons globally (~/.local/share/uv/python/), so the same interpreter is reused across all senselab
  subprocess venvs.

Test plan

• Reproduce the original CUDA-tag mismatch on a clean machine: delete ~/.cache/senselab/venvs/nemo-canary-qwen, run a script that calls transcribe_audios with a Canary-Qwen
  model, confirm it fails on import torchaudio.
• Apply this PR, repeat: confirm the install succeeds and the venv ends up with torch + torchaudio both at +cu128.
• Verify ls ~/.cache/senselab/venvs/nemo-canary-qwen/lib/python3.12/site-packages/ | grep -E "^(torch|torchaudio)-[0-9].*dist-info$" shows both wheels at +cu128.
• Confirm second invocation hits the fixup marker and skips the reinstall (sub-second ensure_venv reuse path).
• Confirm cdifflib build issue is gone on at least one spack-Python host (MIT ORCD).
• Confirm Sortformer / Conformer-CTC venvs still build (touched the shared ensure_venv for --managed-python).

Diff size

2 files changed, 37 insertions(+), 2 deletions(-) — 1 line in subprocess_venv.py, ~28 in canary_qwen.py (the gated fixup block + multi-line import).

[satra]

@wilke0818 - did you see #516

[gemini-code-assist]

Code Review

This pull request implements a CUDA version fixup for the Canary-Qwen speech-to-text task to resolve a runtime mismatch between Torch and TorchAudio, and updates the virtual environment creation to use uv's --managed-python flag. Feedback indicates that the fixup logic is susceptible to race conditions in multi-process environments and recommends using a file lock for synchronization. Additionally, it is suggested to allow subprocess output to be visible to the user for better progress tracking during the large download.

[gemini-code-assist]

The CUDA fixup block is susceptible to a race condition. Since ensure_venv releases its lock before this block executes, multiple concurrent processes might attempt to run the uv pip install command simultaneously if the marker file hasn't been created yet. This could lead to virtual environment corruption or redundant heavy downloads in multi-process environments.

Additionally, using capture_output=True hides progress and error details for a potentially slow operation (downloading ~600MB of wheels). It is better to let the output stream to the console so the user can see progress and diagnose failures easily.

        if not cuda_fixup_marker.is_file():
            from filelock import FileLock
            lock_path = venv_dir.parent / f"{venv_dir.name}.lock"
            with FileLock(str(lock_path), timeout=600):
                # Re-check marker inside the lock to prevent race conditions
                if not cuda_fixup_marker.is_file():
                    subprocess.run(
                        [
                            _find_uv(),
                            "pip",
                            "install",
                            "--python",
                            venv_python(venv_dir),
                            "--force-reinstall",
                            "--no-deps",
                            "--index-url",
                            "https://download.pytorch.org/whl/cu128",
                            "torch>=2.8,<2.9",
                            "torchaudio>=2.8,<2.9",
                        ],
                        check=True,
                    )
                    cuda_fixup_marker.touch()

[wilke0818]

@wilke0818 - did you see #516

nope. will test that one instead

[wilke0818]

use #516 instead