modified_beam_search with NeMo TDT (Parakeet) hallucinates or returns empty text ~20% of the time; greedy_search works

[dfmore]

Disclosure: This issue was filed by an AI coding agent (Claude Code, by Anthropic) on behalf of a developer. The technical details, reproduction steps, and analysis are based on real-world testing in a production Electron application.

Summary

When using modified_beam_search with the parakeet-tdt-0.6b-v3 model (NeMo transducer) via the offline WebSocket server binary, approximately 20% of transcription requests return either:

• The hallucinated text "Yeah." (on audio containing clear speech), or
• Empty text (on audio with good signal, RMS ~0.15)

Switching to greedy_search (by omitting --decoding-method) on the same audio produces correct transcriptions every time.

Environment

• sherpa-onnx version: v1.12.25 (pre-built Windows x64 binary)
• Binary: sherpa-onnx-ws-win32-x64.exe (offline WebSocket server)
• Model: parakeet-tdt-0.6b-v3 (encoder.int8.onnx, decoder.int8.onnx, joiner.int8.onnx)
• OS: Windows 11 x64
• Hardware: CPU inference (no CUDA)

Server launch command (failing)

sherpa-onnx-ws-win32-x64.exe \
  --encoder=encoder.int8.onnx \
  --decoder=decoder.int8.onnx \
  --joiner=joiner.int8.onnx \
  --tokens=tokens.txt \
  --port=6006 \
  --decoding-method=modified_beam_search \
  --hotwords-file=hotwords.txt \
  --hotwords-score=1.5 \
  --modeling-unit=bpe \
  --bpe-vocab=bpe.vocab

Server launch command (working)

sherpa-onnx-ws-win32-x64.exe \
  --encoder=encoder.int8.onnx \
  --decoder=decoder.int8.onnx \
  --joiner=joiner.int8.onnx \
  --tokens=tokens.txt \
  --port=6006

(Defaults to greedy_search — works correctly on all test audio.)

Reproduction steps

1. Download parakeet-tdt-0.6b-v3 model files
2. Create hotwords.txt with a few English words (e.g., one word per line — content doesn't matter, see below)
3. Start the WS server with the "failing" command above
4. Send 30+ transcription requests via WebSocket with clear English speech audio (16kHz mono float32)
5. Observe: ~6/30 return "Yeah.", ~4/30 return empty, ~20/30 return correct text
6. Restart server with the "working" command (no --decoding-method flag)
7. Repeat step 4 with the same audio files
8. Observe: 30/30 return correct transcriptions

Key observations

• Not caused by hotwords content: Testing with an empty hotwords file + --hotwords-score=0.2 still produces hallucinations. The trigger is --decoding-method=modified_beam_search itself.
• Not caused by audio quality: The same audio that returns "Yeah." with MBS returns perfect transcription with greedy search. Audio RMS is ~0.15 (not silence).
• Not caused by warm-up state: A 1-second silence warm-up inference runs before test trials. Hallucinations persist after warm-up.
• Intermittent: The same audio file may succeed on one request and hallucinate on the next. No deterministic pattern.
• Affects the NeMo TDT decoder specifically: This uses the OfflineTransducerModifiedBeamSearchNeMoDecoder code path added in PR Add modified beam search and hotwords support for NeMo transducer models #3077.

Likely related issues

• Decoding method 'modified_beam_search' gives letters/words on silence, while 'greedy_search' works well #845: "Decoding method 'modified_beam_search' gives letters/words on silence" — same symptom (MBS hallucinates, greedy works) but on a custom zipformer model, not NeMo TDT. The maintainer was unable to identify the root cause. Our report may share the same underlying bug in the MBS decoder.
• Why parakeet (transducer model) can't support hotwords? #2541 / PR Add modified beam search and hotwords support for NeMo transducer models #3077: Added modified_beam_search support for NeMo transducer models (merged Feb 5, 2026). The NeMo MBS decoder is relatively new and may have less field testing.

Workaround

We default to greedy_search and only enable modified_beam_search when the user explicitly opts in to hotword boosting (with a warning that it's experimental). This eliminates the hallucination for the default path.

WebSocket protocol used

Standard sherpa-onnx offline WS binary protocol:

[int32LE sample_rate][int32LE num_audio_bytes][float32 samples...]

Response: JSON with text field. After receiving response, client sends "Done" and connection closes.

What we'd like

1. Confirmation of whether this is a known limitation of the NeMo MBS decoder
2. If it's a bug, a fix so that modified_beam_search + hotwords works reliably with NeMo TDT models
3. Any recommended --hotwords-score values or configuration that might reduce the hallucination rate

Thank you for sherpa-onnx — it's an excellent project and Parakeet TDT transcription quality with greedy_search is outstanding.

[dfmore]

Update from the original reporter (2026-03-18):

After further investigation, I want to clarify the scope of this issue:

1. This issue (modified_beam_search with NeMo TDT (Parakeet) hallucinates or returns empty text ~20% of the time; greedy_search works #3267) is specifically about modified_beam_search — greedy_search works correctly, as stated in the title. We have confirmed this in production.

2. The actual greedy_search empty-text issue is Empty transcription with "parakeet-tdt-0.6b-v2" on some Files #2258 (dither=0 causing numerical edge cases in the log-mel filterbank). We have just added --dither=0.00003 to our server launch command as a fix.

3. The modified_beam_search hallucinates/empty issue (modified_beam_search with NeMo TDT (Parakeet) hallucinates or returns empty text ~20% of the time; greedy_search works #3267) remains unresolved in our testing. We continue to use greedy_search + JS post-processing for dictionary substitution as a workaround.

We'll report back after testing --dither=0.00003 to confirm whether it resolves the greedy_search empty-output cases we observed.

[kittysoftpaw0510]

Fix submitted in #3657.