Add NVIDIA Canary-1b-v2 ASR backend

README.md

@@ -124,6 +124,7 @@ For a native SwiftUI macOS client, see [macos/WhisperLiveKitMac](macos/WhisperLi
 | **Qwen3-ASR vLLM Metal (Apple Silicon)** | Install vLLM with the official vllm-metal script first, then `uv sync --extra qwen3-vllm-metal` | Install vLLM with the official vllm-metal script first, then `pip install -e ".[qwen3-vllm-metal]"` |
 | **Speaker diarization (Sortformer / NeMo)** | `uv sync --extra diarization-sortformer` | `pip install -e ".[diarization-sortformer]"` |
 | *[Not recommended]* Speaker diarization with Diart | `uv sync --extra diarization-diart` | `pip install -e ".[diarization-diart]"` |
+| **Canary-1b-v2 (NeMo, CUDA)** | `uv sync --extra canary` | `pip install -e ".[canary]"` |
 
 Supported GPU profiles:
 
@@ -138,7 +139,7 @@ uv sync --extra cu129 --extra voxtral-hf --extra translation
 uv sync --extra qwen3-vllm
 ```
 
-`qwen3-vllm` uses vLLM's CUDA wheel stack and must be installed in a separate environment from `cu129`. `voxtral-hf` / `qwen3-vllm-metal` and `diarization-sortformer` are also intentionally incompatible extras and must be installed in separate environments.
+`qwen3-vllm` uses vLLM's CUDA wheel stack and must be installed in a separate environment from `cu129`. `voxtral-hf` / `qwen3-vllm-metal` / `canary` and `diarization-sortformer` are also intentionally incompatible extras and must be installed in separate environments.
 
 See **Parameters & Configuration** below on how to use them.
 
@@ -263,6 +264,28 @@ text-decoder request per chunk. The `append-kv` and `rolling` names remain as
 compatibility aliases for the HF decoder path. Keep standard `qwen3-vllm` for
 best current accuracy until the causal quality gate is fixed.
 
+### Canary Backend
+
+WhisperLiveKit supports [NVIDIA Canary-1b-v2](https://huggingface.co/nvidia/canary-1b-v2)
+via [NeMo](https://github.com/NVIDIA/NeMo), a 1B-parameter model covering 25 European
+languages with native word-level timestamps. Automatic language detection uses NeMo's
+AmberNet language-ID model when `--language auto` is set; the detected language is locked
+in once enough audio has accumulated. Canary streams through the LocalAgreement policy.
+
+```bash
+pip install -e ".[canary]"
+wlk --backend canary --language auto
+```
+
+Notes:
+- CUDA only; NeMo is a heavy dependency (torch, pytorch-lightning, and friends).
+- Word/segment timestamps require NeMo's timestamp API, available in NeMo 2.5+.
+  The `canary` extra pins `nemo-toolkit[asr]>=2.5.0`; if you land on a build where
+  the timestamp API is missing, install NeMo from `main`.
+- Explicit `--language <code>` skips language detection entirely. Tune detection
+  with `--canary-lid-min-sec` (minimum audio before detecting) and
+  `--canary-lid-min-conf` (confidence threshold to lock in the detected language).
+
 ### Usage Examples
 
 **Command-line Interface**: Start the transcription server with various options:
@@ -334,7 +357,7 @@ async def websocket_endpoint(websocket: WebSocket):
 | `--translation-backend` | `nllb` (in-process, CPU-friendly) or `alignatt`: streaming LLM translation through an [Alignatt4LLM](https://github.com/QuentinFuxa/Alignatt4LLM) sidecar, with attention-gated append-only commits. See [docs/translation-alignatt.md](docs/translation-alignatt.md). | `nllb` |
 | `--diarization` | Enable speaker identification | `False` |
 | `--backend-policy` | Streaming strategy: `1`/`simulstreaming` uses AlignAtt SimulStreaming, `2`/`localagreement` uses the LocalAgreement policy | `simulstreaming` |
-| `--backend` | ASR backend selector. `auto` picks MLX on macOS (if installed), otherwise Faster-Whisper, otherwise vanilla Whisper. Options: `mlx-whisper`, `faster-whisper`, `whisper`, `openai-api` (LocalAgreement only), `voxtral-mlx` (Apple Silicon), `voxtral` (HuggingFace), `qwen3-vllm`, `qwen3-vllm-metal` (Apple Silicon), `qwen3-streaming` (HuggingFace, CUDA/MPS/CPU) | `auto` |
+| `--backend` | ASR backend selector. `auto` picks MLX on macOS (if installed), otherwise Faster-Whisper, otherwise vanilla Whisper. Options: `mlx-whisper`, `faster-whisper`, `whisper`, `openai-api` (LocalAgreement only), `voxtral-mlx` (Apple Silicon), `voxtral` (HuggingFace), `qwen3-vllm`, `qwen3-vllm-metal` (Apple Silicon), `qwen3-streaming` (HuggingFace, CUDA/MPS/CPU), `canary` (NeMo, CUDA) | `auto` |
 | `--no-vac` | Disable Voice Activity Controller. NOT ADVISED | `False` |
 | `--no-vad` | Disable Voice Activity Detection. NOT ADVISED | `False` |
 | `--warmup-file` | Audio file path for model warmup | `jfk.wav` |
@@ -360,6 +383,14 @@ async def websocket_endpoint(websocket: WebSocket):
 | `--segmentation-model` | Hugging Face model ID for Diart segmentation model. [Available models](https://github.com/juanmc2005/diart/tree/main?tab=readme-ov-file#pre-trained-models) | `pyannote/segmentation-3.0` |
 | `--embedding-model` | Hugging Face model ID for Diart embedding model. [Available models](https://github.com/juanmc2005/diart/tree/main?tab=readme-ov-file#pre-trained-models) | `pyannote/embedding` |
 
+| Canary backend options (only used with `--backend canary`) | Description | Default |
+|-----------|-------------|---------|
+| `--canary-model` | HuggingFace/NGC model id or local `.nemo` path | `nvidia/canary-1b-v2` |
+| `--canary-default-lang` | Language used until auto-detection locks in (or always, with explicit `--language`) | `en` |
+| `--canary-lid-model` | NeMo language-ID model used for `--language auto` | `langid_ambernet` |
+| `--canary-lid-min-sec` | Minimum seconds of audio before attempting language detection | `2.0` |
+| `--canary-lid-min-conf` | Confidence threshold to lock in the detected language | `0.5` |
+
 | SimulStreaming backend options | Description | Default |
 |-----------|-------------|---------|
 | `--disable-fast-encoder` | Disable Faster Whisper or MLX Whisper backends for the encoder (if installed). Inference can be slower but helpful when GPU memory is limited | `False` |

pyproject.toml

@@ -72,6 +72,9 @@ cu129 = [
 diarization-sortformer = [
     "nemo-toolkit[asr]>2.4; python_version >= '3.10' and python_version < '3.13'",
 ]
+canary = [
+    "nemo-toolkit[asr]>=2.5.0; python_version >= '3.10' and python_version < '3.13'",
+]
 diarization-diart = [
     "diart",
     "torch<2.9.0",
@@ -128,6 +131,14 @@ conflicts = [
         { extra = "qwen3-streaming" },
         { extra = "voxtral-hf" },
     ],
+    [
+        { extra = "canary" },
+        { extra = "voxtral-hf" },
+    ],
+    [
+        { extra = "canary" },
+        { extra = "qwen3-vllm-metal" },
+    ],
 ]
 
 [tool.uv.sources]

scripts/smoke_canary.py

@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+"""Smoke-test the NVIDIA Canary-1b-v2 ASR backend — no GPU required.
+
+Loads the Canary backend (NeMo ``EncDecMultiTaskModel``), transcribes a short
+16 kHz mono clip, and prints the text plus per-word timestamps. Runs on CPU
+(slower, but fully functional) so the backend can be validated without a GPU.
+Exits non-zero if the model produces no output.
+
+Model card: https://huggingface.co/nvidia/canary-1b-v2
+
+Usage
+-----
+    pip install -e ".[canary]"
+
+    # Validate with your own 16 kHz mono clip:
+    python scripts/smoke_canary.py path/to/clip.wav
+
+    # Or with no argument, using a bundled LibriSpeech test sample:
+    python scripts/smoke_canary.py
+
+The model (~4 GB) downloads from Hugging Face on first run. For an equivalent
+in-pipeline check, see the ``TestHarness`` snippet in tests/test_canary_backend.py
+(``test_canary_end_to_end_via_testharness``).
+"""
+
+import argparse
+import sys
+
+import numpy as np
+
+
+def load_audio(path: str) -> np.ndarray:
+    """Load an audio file as a 16 kHz mono float32 numpy array."""
+    import soundfile as sf
+
+    audio, sr = sf.read(path, dtype="float32")
+    if audio.ndim > 1:
+        audio = audio.mean(axis=1)
+    if sr != 16000:
+        import librosa
+
+        audio = librosa.resample(audio, orig_sr=sr, target_sr=16000)
+    return np.asarray(audio, dtype=np.float32)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description=__doc__,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    parser.add_argument(
+        "audio",
+        nargs="?",
+        help="16 kHz mono WAV/FLAC path. If omitted, a bundled LibriSpeech sample is used.",
+    )
+    parser.add_argument(
+        "--model",
+        default="nvidia/canary-1b-v2",
+        help="Canary model id or local .nemo path (default: nvidia/canary-1b-v2).",
+    )
+    parser.add_argument(
+        "--language",
+        default="en",
+        help="Source language code, e.g. en, fr, de (default: en).",
+    )
+    args = parser.parse_args()
+
+    reference = None
+    if args.audio:
+        audio = load_audio(args.audio)
+    else:
+        from whisperlivekit.test_data import get_sample
+
+        sample = get_sample("librispeech_short")
+        audio = load_audio(sample.path)
+        reference = getattr(sample, "reference", None)
+
+    from whisperlivekit.canary_backend import CanaryASR
+
+    print(f"Loading Canary backend '{args.model}' (CPU is fine; first run downloads ~4 GB)...")
+    asr = CanaryASR(lan=args.language, canary_model=args.model)
+
+    res = asr.transcribe(audio, source_lang=args.language)
+    tokens = asr.ts_words(res)
+    # Canary emits space-prefixed word tokens (sep=""), so a plain join is correct.
+    text = "".join(t.text for t in tokens).strip()
+
+    print("\n--- transcription ---")
+    print(text or "(empty!)")
+    if reference:
+        print(f"\n(reference: {reference})")
+
+    print("\n--- word timestamps (first 12) ---")
+    for t in tokens[:12]:
+        print(f"  [{t.start:6.2f} -> {t.end:6.2f}] {t.text!r}")
+
+    ok = bool(text)
+    print(f"\nSMOKE {'PASS' if ok else 'FAIL'}: {len(tokens)} word token(s)")
+    sys.exit(0 if ok else 1)
+
+
+if __name__ == "__main__":
+    main()