fix: keep ONNX models from dropping whole recordings - #256
Conversation
The specialized ONNX models silently returned nothing for perfectly good audio, at any length and seemingly at random. The user-visible effect is a recording that transcribes to nothing at all. Cause: sherpa-onnx's NeMo-derived decoders (Canary here, same decoder family elsewhere) run greedy attention decoding and stop as soon as they emit end-of-transcript. When speech begins in sample zero — which push-to-talk recording makes common, since capture starts on the first frame the tap delivers — the model reads the clip as an utterance already in progress and emits end-of-transcript as its *first* token. sherpa-onnx then pops that token and hands back an empty string rather than an error, so nothing downstream can tell the recording was dropped. Give every segment 200ms of silence on each side before it reaches native code. Reproduced against Canary 180M with clips that returned "" — a lead-in as short as 50ms recovers all of them; 200ms leaves margin, and the matching tail keeps a final word from being cut off mid-decode. Verified across 14 clips: every previously empty one now transcribes and the ones that already worked are unchanged. The recording's reported duration is measured before padding, so it is unaffected. Also log a warning when a decode still comes back empty. The old INFO line read "Result: ..." either way, which is why this went unnoticed in the logs for so long.
✅ Deploy Preview for voca-mac canceled.
|
|
| Filename | Overview |
|---|---|
| Sources/VocaMac/Services/SherpaAudioPreparation.swift | Adds symmetric edge silence while preserving the minimum sample count and bypassing digital silence. |
| Sources/VocaMac/Services/SherpaService.swift | Accounts for padding in segment limits and correctly separates skipped silence from unexpected empty decodes. |
| Tests/VocaMacTests/SherpaAudioPreparationTests.swift | Covers edge padding, minimum length, model-limit compatibility, and preservation of original samples. |
| Tests/VocaMacTests/SherpaServiceTests.swift | Updates segment-processing expectations to account for symmetric padding. |
Reviews (3): Last reviewed commit: "Merge branch 'main' into fix/onnx-empty-..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Two follow-ups from review. The lead-in and tail count against the same length limit the segmenter is there to respect: capping a segment at maxSegmentSeconds and then adding 400ms sent up to 8.4s to a model documented to degrade past 8s, which is the failure this change set out to fix. Take the padding out of the segment budget so what reaches the decoder stays inside the limit, and pin that with a test over every model in the catalog. Digital silence is skipped before native inference, so reporting it as a decode that returned nothing blurs the very signal the new warning exists to make obvious. Log it as the no-op it is. Both the GUI and the CLI already reject silence upstream, so this is reachable only through direct service calls today.
|
Both Greptile findings were real. Fixed in 321c1b0. P1 — padding exceeds decoder limits: true, and the more serious of the two.
The segment budget now subtracts what let segmentLimit = SherpaModelCatalog.spec(for: size)?.maxSegmentSeconds
let maxSeconds = segmentLimit.map { $0 - SherpaAudioPreparation.addedSilenceSeconds }Moonshine/SenseVoice get a 7.6s speech budget, the NeMo models 19.6s. The trade is that clips within 400ms of a limit now split where they previously ran in one pass — a join is the normal path for anything longer anyway, and it beats overrunning a limit the catalog documents as "returns nothing at all". Two tests pin it: one asserts P2 — warning mislabels skipped silence: true, but not reachable in production. Correct that digital silence never reaches native inference, so calling it a decode that returned nothing muddies the exact signal the warning exists to provide. It now logs as the no-op it is. Worth noting for reviewers that the branch is unreachable through today's entry points: Verification: |
I am glad we are finding greptile useful. |
* feat: save the audio behind an ONNX decode that returns nothing An empty decode is the one failure with nothing to debug: no error, no text, and the audio is gone the moment the buffer is released. The bug in #256 took two wrong fixes before a dumped recording showed what was actually happening — the decode turned out to be so sensitive to its input that scaling the same samples by 1.001 changed the result, which no synthetic clip reproduced and no log line could have revealed. Write the samples to a WAV next to the logs so the failure can be replayed through --transcribe-file. Off unless asked for, since it puts recorded speech on disk: defaults write com.vocamac.app vocamac.debug.saveFailedAudio -bool true Keeps the newest 20 recordings so it cannot fill the disk, and only fires for audio that actually reached the decoder — digital silence is skipped before inference and is not a failure. * Avoid colliding failed-audio dump filenames Include a short UUID token so two empty ONNX dumps in the same second keep both WAV files. Sanitize model path characters too. --------- Co-authored-by: Jatin K Malik <jatinkrmalik@gmail.com>
2 participants
The bug
The specialized ONNX models sometimes transcribe a recording to nothing at all — no text injected, no error. It looked random: it hit clips of 3s, 50s and a minute alike, with no pattern in the audio.
Root cause
It is not random, and it is not about length or language.
sherpa-onnx's NeMo-derived decoders (Canary here; the same decoder family elsewhere) run greedy attention decoding and stop the moment they emit
<|endoftext|>:When speech begins in sample zero, the model reads the clip as an utterance already in progress and emits end-of-transcript as its first token.
tokensis then just{eos},pop_back()empties it, and sherpa-onnx returns an empty string rather than an error — so nothing downstream can tell the recording was dropped. Push-to-talk makes this common: capture starts on the first frame the tap delivers, so whether a clip has a lead-in is luck.The signature in the logs is a decode that finishes far too fast:
Reproduced with the headless CLI against Canary 180M: an 8.7s English clip returned
"", and truncations of that same file alternated between good text and empty at 3s / 5s / 7s / 8s.The fix
Give every segment 200ms of silence on each side before it reaches native code (
SherpaAudioPreparation.prepare).A lead-in as short as 50ms recovers every clip that failed this way; 200ms leaves margin, and the matching tail keeps a final word from being cut off mid-decode. The recording's reported duration is measured from the original samples before padding, so
audioLengthSecondsis unaffected.Also log a warning when a decode still comes back empty. The old INFO line read
Result: ...whether or not there was a result, which is why this went unnoticed in the logs for so long.Verification
14 clips through the built app's
--transcribe-file, before and after:""""""Every previously empty clip now transcribes; nothing that already worked regressed.
swift test: 474 tests, 0 failures. Adds a regression test that speech never starts in the first sample, and updates the two existing tests whose expected sample counts shift by the padding.Notes