[speechlm2] Add streaming inference pipeline for NemotronVoiceChat#15571

Open

erastorgueva-nv wants to merge 45 commits intoNVIDIA-NeMo:mainfrom

erastorgueva-nv:duplex-realtime-inference-rebase

Commits on Apr 1, 2026

add changes from duplex-realtime-inference branch, except duplex_stt_model.py modification for function_head
erastorgueva-nv
committed
add on: asr_logits boosts, speaker embedding, fc head
erastorgueva-nv
committed
add use_llm_cache option, will use HybridMambaAttentionDynamicCache, with patches
erastorgueva-nv
committed
add tts inference speedups: vectorize depthsum, precompute rvq schedule, optional torch.compile & subword cache
erastorgueva-nv
committed
allow using speaker_latent with vllm (need to update vllm eartts.py)
erastorgueva-nv
committed
add flag for speaker_name if doing standalone inference
erastorgueva-nv
committed
remove standalone code path; add parity check for offline vs streaming - adjusted infer_one_step code so operations will match offline
erastorgueva-nv
committed
skip pretrained ASR/LLM downloads in from_pretrained; simplify inference wrapper loading
erastorgueva-nv
committed
quickfix for parity harness regarding speaker name / reference in tts
erastorgueva-nv
committed
speed up model loading: use meta device, dont get codec silence tokens which will be ignored anyway
erastorgueva-nv
committed
normalize indentation to 4-space
erastorgueva-nv
committed
remove hardcoded env var, simple tidy: remove dead code atc
erastorgueva-nv
committed
always use codec cache => remove use_codec_cache flag and codec_token_history_size parameter
erastorgueva-nv
committed
remove newlines in logs
erastorgueva-nv
committed
further tidying: pass StreamingDecodeState directly, return InferenceStepResult etc
erastorgueva-nv
committed
Add pytest-based offline vs. incremental inference parity test with logit comparison
erastorgueva-nv
committed
refactor streaming S2S pipeline: extract helpers, factor infer_one_step, add docs
erastorgueva-nv
committed
in test: use existing audio file, allow system prompt, specify params for parity
erastorgueva-nv
committed
Refactor voicechat tests: shared fixtures, no-crash sweep, deterministic parity
erastorgueva-nv
committed
Fix byte-level BPE decoding in raw output: unify tokens_to_str and tokens_to_str_raw
erastorgueva-nv
committed
use whisper normalizer for wer calculation
erastorgueva-nv
committed
remove unnecessary logging in perception cache step
erastorgueva-nv
committed
vectorize rep penalty; fix sampling - nan/inf check before top-p filtering
erastorgueva-nv
committed
Preserve BOS/EOS as literal strings in decoded text output
erastorgueva-nv
committed
update triton code; bugfix for vllm dtype/device
erastorgueva-nv
committed
Always send prefill before audio streaming; fix bfloat16 audio output
erastorgueva-nv
committed
remove triton code to keep PR simple
erastorgueva-nv
committed
add missing __init__.py
erastorgueva-nv
committed
use built-in type hints (X | None, dict, list) instead of typing imports
erastorgueva-nv
committed
use nemo_asr.metrics.wer.word_error_rate for wer calc
erastorgueva-nv
committed
use SimpleTimer in s2s_streaming_infer.py script
erastorgueva-nv
committed

Commits on Apr 9, 2026

refactor: split model_factory into backend/ modules; unify vLLM engine classes
erastorgueva-nv
committed

Commits on Apr 10, 2026

Commits on Apr 14, 2026

Commits on Apr 15, 2026

fix O(n^2) output accumulation, release state when stream finished, make ctm token-level
erastorgueva-nv
committed

Commits on Apr 16, 2026

Rename S2SStreamingState to S2SStreamingOutput, remove slots & PipelineOutput, move token finalization into output object
erastorgueva-nv
committed

Commits on Apr 17, 2026