Add NVIDIA Canary-1b-v2 ASR backend

[wuxuedaifu]

Summary

Adds a canary ASR backend running NVIDIA Canary-1b-v2 on the existing LocalAgreement streaming policy (same mechanism as the Whisper backends). Canary-1b-v2 is a ~1B-param attention encoder-decoder (FastConformer encoder + Transformer decoder) loaded via NeMo's EncDecMultiTaskModel; it emits native word/segment timestamps, covers 25 European languages, and runs on CUDA or CPU.

⚠️ Exact model id (re: your "Canny-qwen-1b" note)

The model is nvidia/canary-1b-v2 — model card: https://huggingface.co/nvidia/canary-1b-v2 (paper: https://arxiv.org/abs/2509.14128). There is no "Canny-qwen-1b"; that looks like a mix-up of Canary and Qwen. This backend is NVIDIA Canary 1B v2 only, unrelated to the Qwen3 backends. The id is the default everywhere (--canary-model, config.canary_model).

Follows CLAUDE.md → "Adding a New ASR Backend"

1. whisperlivekit/canary_backend.py — CanaryASR implements transcribe(audio, init_prompt=""), ts_words(result), segments_end_ts(result), use_vad().
2. Required attributes set on the class: sep, original_language, backend_choice, SAMPLING_RATE, confidence_validation, tokenizer, buffer_trimming, buffer_trimming_sec.
3. Registered in core.py — an elif config.backend == "canary" branch in TranscriptionEngine._do_init() (before the backend_policy branch), and a routing case in online_factory() returning OnlineASRProcessor over a per-session wrapper.
4. CLI — "canary" added to --backend choices in parse_args.py, plus a --canary-* option group.

The module lazy-imports NeMo/torch, so it imports fine (and CI runs) without nemo_toolkit installed.

Extra pieces

• CanaryLID (NeMo langid_ambernet) drives optional auto language detection when --language auto, via a CanarySessionASR (a SessionASRProxy subclass) that detects once on the first speech then locks the session language. If the LID model can't be loaded (e.g. offline), it degrades to --canary-default-lang instead of failing startup.
• pyproject.toml: a canary optional extra (nemo-toolkit[asr], marker matching the sibling diarization-sortformer) with [tool.uv] conflict entries vs voxtral-hf / qwen3-vllm-metal (NeMo's transformers pin conflicts with those); uv.lock regenerated accordingly (large but auto-generated — the churn is uv's conflict-marker encoding, only +2 packages).
• README: install, --backend canary, and the --canary-* flags documented alongside the other backends.

Validating without a GPU (per your ask)

Canary runs on CPU. Install and run the included smoke script:

pip install -e ".[canary]"          # heavy: NeMo + torch; needs Python 3.10–3.12
python scripts/smoke_canary.py path/to/16k_mono.wav
# or, no arg -> uses the bundled LibriSpeech test sample
python scripts/smoke_canary.py

Real output from a CPU/MPS run (macOS, no CUDA), transcribing a TTS clip of "the quick brown fox jumps over the lazy dog":

INFO:whisperlivekit.canary_backend:Canary model loaded in 16.79s
--- transcription ---
as the quick brown fox jumps over the lazy dog.
--- word timestamps (first 12) ---
  [  0.00 ->   0.08] ' as'
  [  0.16 ->   0.24] ' the'
  [  0.24 ->   0.40] ' quick'
  ...
  [  2.16 ->   2.24] ' dog.'
SMOKE PASS: 10 word token(s)

Equivalent in-pipeline check via TestHarness (gated behind NeMo, skips when absent — tests/test_canary_backend.py::test_canary_end_to_end_via_testharness):

import asyncio
from whisperlivekit import TestHarness
from whisperlivekit.test_data import get_sample

async def main():
    sample = get_sample("librispeech_short")
    async with TestHarness(backend="canary", lan="en") as h:
        await h.feed(sample.path, speed=1.0)
        await h.drain(3.0)
        result = await h.finish()
        print(result.text)          # non-empty transcription

asyncio.run(main())

I also ran it live end-to-end through the /asr WebSocket UI on this branch (mic → FFmpeg → VAD → Canary → LocalAgreement), transcribing correctly and near-real-time on MPS.

Test plan

• pytest tests/test_canary_backend.py → 14 passed, 2 skipped (the 2 skips are the NeMo-gated real-model tests). Covers helpers, VoxLingua→Canary code mapping, the CanarySessionASR detect-once-then-lock state machine (incl. low-confidence / LID-exception / no-LID paths), and online_factory routing. Runs in CI without NeMo.
• Smoke script + live /asr run validated on CPU/MPS (above).
• Please run scripts/smoke_canary.py on your side to confirm on your hardware.

Notes / caveats

• Word/segment timestamps require NeMo's timestamp API (NeMo 2.5+); if a pinned release lags on it, install NeMo from main.
• Auto-detect needs langid_ambernet reachable; otherwise it falls back to --canary-default-lang (fail-soft).
• Translation (AST) via Canary's target_lang is intentionally out of scope here; a native AlignAtt streaming variant could be a later follow-up.

Happy to adjust anything to fit the project's conventions.

[QuentinFuxa]

Thanks for correcting the model id and for following the backend guide; the adapter structure is right and the smoke script is a good addition. Review of the current head:

Blockers:

1. CI lint is red: ruff flags E402 twice in tests/test_canary_backend.py (the import importlib.util and import pytest at lines 194/196 must move to the top of the file; the _NEMO_AVAILABLE/requires_nemo computation can stay where it is).
2. Language auto-detection looks wired to a config path NeMo does not expose: CanaryLID will not find the label list where it reads it (NeMo keeps it at model._cfg['train_ds'].labels), so --language auto likely never detects and silently falls back. Please verify against a real checkpoint and make the no-labels case loud.

Asks:
3. A per-session ?language= outside CANARY_LANGS yields a silently dead session; validate it and surface the error like the config-time check does.
4. Missing NeMo raises a bare ModuleNotFoundError; wrap it with an actionable install hint like the other optional backends.
5. The README claims the canary extra is environment-incompatible with diarization-sortformer, but the two resolve together fine; drop or justify that claim. Also, the README says CUDA only while the PR body and the smoke script claim CPU validation; make them agree.
6. On Python 3.13 the canary extra resolves to nothing (NeMo's python pin), so users there get a confusing no-op install; add a config-time error or a clear README note.
7. Typography: several added comments and docs use em dashes; this repo does not use them, please replace with commas or colons.

On validation: every model-dependent test is NeMo-gated and skips in CI, so the accuracy and CPU claims rest entirely on your local runs. Before merging a 1B-model backend I would like one reproducible trace: please paste the full output of scripts/smoke_canary.py at the current head from a machine you control, with the exact command line.

[QuentinFuxa]

Gentle ping on the July 22 review. The two blockers (ruff E402 in the test file, and the CanaryLID label-list path) are the only things standing between this and a merge; the rest can land as follow-ups if you prefer. Happy to re-review quickly once you push.

[wuxuedaifu]

Pushed 6c7a505 addressing the review.

Blockers

1. ruff E402 — hoisted importlib.util and pytest to the top of tests/test_canary_backend.py; left the _NEMO_AVAILABLE/requires_nemo computation where it was. ruff check is clean.

2. CanaryLID label path — you were right to flag this, and it was worse than it looked: the langid_ambernet download had failed during my original live testing, so self.model.cfg.labels was never actually exercised against a real checkpoint. Rather than blind-swap to cfg.train_ds.labels, I added resolve_lid_labels() which probes both known NeMo layouts (cfg.labels, then cfg.train_ds.labels), resolves eagerly in CanaryLID.__init__ (so a checkpoint we can't map fails at load — where core.py already fail-softs to disabling auto-detect — instead of silently mispredicting on the first detect()), and raises a clear RuntimeError if neither exists. detect() now reads the resolved self.labels.

Honest caveat: I could not verify this against a real ambernet checkpoint. Its NeMo cache on my machine is empty (0 B — the original download failed) and NGC is unreachable on my current network. The fix is written to tolerate either config path and to fail loudly on neither; I'd appreciate a sanity check from anyone with a working langid_ambernet as to which path the shipped checkpoint actually uses.

Asks

3. Per-session ?language= (and config-time --language/--canary-default-lang) outside CANARY_LANGS now surface an error instead of a silently dead session: a ValueError in CanarySessionASR.__init__, a config-time check in core.py, and a client-facing websocket.close(4400, reason=...) in basic_server.py.
4. Both NeMo imports are wrapped with an actionable ImportError (_NEMO_INSTALL_HINT) that also names the Python 3.10–3.12 pin.
5. README: dropped the incorrect "canary is incompatible with diarization-sortformer" claim — they co-resolve fine (both pull nemo-toolkit[asr]), and [tool.uv].conflicts only lists canary against voxtral-hf/qwen3-vllm-metal. Also changed "CUDA only" → "CUDA/MPS/CPU" in all three spots to match the smoke script.
6. Added a README note + import-hint line that the canary extra requires Python 3.10–3.12 and no-ops on 3.13+.
7. Removed the canary-added em dashes (left pre-existing framework ones untouched).

Added 4 non-NeMo-gated tests (label resolver + language rejection). 18 passed, 2 NeMo-gated skipped.

Requested trace

Isolated venv, NeMo 2.7.3 / torch 2.13 / Python 3.12, Darwin arm64, real 1B model from HF cache. I used a local clip because the bundled LibriSpeech sample download hits an SSL block on my network — the script's positional path argument is meant for exactly this:

$ python scripts/smoke_canary.py third_party/qwen3-asr-causal/tests/data/e2e_smoke.wav --language en
INFO:whisperlivekit.canary_backend:Loading Canary model 'nvidia/canary-1b-v2' via NeMo...
[NeMo I] Model EncDecMultiTaskModel was successfully restored from .../canary-1b-v2.nemo
INFO:whisperlivekit.canary_backend:Canary model loaded in 21.69s
--- transcription ---
The quick brown fox jumps over the lazy dog. Streaming transcription should recognize this final sentence.
--- word timestamps (first 12) ---
  [  0.16 ->   0.24] ' The'
  [  0.24 ->   0.40] ' quick'
  [  0.48 ->   0.72] ' brown'
  [  0.80 ->   1.04] ' fox'
  [  1.12 ->   1.44] ' jumps'
  [  1.52 ->   1.60] ' over'
  [  1.68 ->   1.76] ' the'
  [  1.84 ->   2.08] ' lazy'
  [  2.24 ->   2.32] ' dog.'
  [  2.88 ->   3.20] ' Streaming'
  [  3.36 ->   3.92] ' transcription'
  [  4.16 ->   4.24] ' should'

SMOKE PASS: 16 word token(s)

The one thing this trace does not cover is the LID/auto-detect path (blocker #2), for the network reason noted above.

[wuxuedaifu]

Gentle ping on the July 22 review. The two blockers (ruff E402 in the test file, and the CanaryLID label-list path) are the only things standing between this and a merge; the rest can land as follow-ups if you prefer. Happy to re-review quickly once you push.

Fixed the issue, kindly check, thanks