AudioProcessor resample silently corrupts long compressed audio (87-min mp3 → garbage transcript, exit 0)

[kiki830621]

Summary

AudioProcessor's resample path silently produces corrupted PCM for long compressed audio files: an 87-minute 44.1 kHz stereo mp3 transcribes to garbage (a single "(music)"-style cue / a 129 KB blob of repeated tokens) with exit code 0 — no error surfaces anywhere. The same pipeline handles short slices of the same file and 16 kHz mono WAV of any length correctly, which is what makes this dangerous: it looks like a model-quality problem, not an audio-decoding one.

Reproduction matrix (WhisperKit 0.18.0, Apple Silicon, macOS 27)

Input	Result
87-min 44.1 kHz stereo mp3	❌ corrupted transcript (garbage tokens), exit 0
first 5-min slice of the SAME mp3	✅ normal transcript
same 87-min audio pre-converted to 16 kHz mono WAV	✅ normal transcript
any-length 16 kHz mono WAV	✅ normal transcript

Audio used: the public Jobs/Gates D5 conversation recording (any long compressed file should do; the failure threshold between 5 and 87 minutes was not bisected).

Suspected mechanism (not bisected upstream)

Two candidate contributors in the 0.18.0 resample path (AudioProcessor.swift around lines 381–450):

1. The path trusts the compressed container's estimated AVAudioFile.length when sizing reads — for VBR mp3 the estimate can diverge from the real frame count.
2. Resampling happens in chunks (the 10 MB chunking introduced in the old WhisperKit issue tracker as "Resample audio files in 10mb chunks"), and the AVAudioConverter appears to be rebuilt per chunk — a fresh converter loses filter state across chunk boundaries, which corrupts the signal cumulatively on long files while staying inaudible/harmless on short ones.

Related (closed) reports in the old tracker were about explicit failures (resampleBuffer capacity, "Failed to process audio buffer" for an mp3); this one is nastier because nothing fails — the decoded PCM is simply wrong.

Workaround we ship

Pre-convert every input to 16 kHz mono WAV via a single long-lived AVAudioConverter before it reaches WhisperKit; the corruption never reproduces through that path. Happy to test a fix against the same 87-minute file.

[ZachNagengast]

@kiki830621 Thanks for the report - do you have a shareable sample audio file that reproduces this issue?

[kiki830621]

@ZachNagengast Thanks for the quick response — and apologies, because after a much deeper investigation I have to walk this report back: I can no longer reproduce the corruption on WhisperKit 0.18.0, including with the exact file that originally triggered it. The "resample silently corrupts long compressed audio" diagnosis does not hold up. Here is what I found, in case the negative result is still useful.

The original trigger file loads byte-perfectly

The file that produced the original garbage transcript is an 87-min 44.1 kHz stereo, 192 kbps CBR mp3 (ID3v2.3, no Xing/LAME header). Running it back through AudioProcessor on 0.18.0:

path	AVAudioFile.length	decoded 16 kHz frames	vs expected (dur × 16000)	NaN
loadAudio(fromPath:) (whole-file)	accurate (5204.9 s)	83,278,367	+0 (exact)	0
loadAudioAsFloatArray(...) (the 10-min chunked+seek path transcribe() uses)	—	83,278,367	+0 (exact)	0

Both paths are byte-identical and the frame count is exactly the expected resample of the full duration — no truncation, no NaN, no divergence. If the resample path were corrupting PCM, this file should corrupt in place. It doesn't.

I also ran a clean-source matrix (public-domain LibriVox audiobook, 65 & 130 min × CBR/VBR × mono/stereo × with/without Xing header × fat ID3v2.3) — all decode correctly. AVAudioFile.length stayed accurate in every case (CoreAudio's mp3 parser recovers the true length even without a Xing header), so the "trusts the estimated length" mechanism I originally suspected never actually bites.

End-to-end also no longer reproduces

Transcribing the original file end-to-end (language zh, temperature fallback on = the original default), reporting only structural stats (no transcript content):

model	segments	coverage	result
tiny	2324	5205 s	coherent
large-v3	1925	5204 s	coherent

The large-v3 run gives 1925 cues over the full 5204 s — which matches the good cell of my original report (the 16 kHz-mono WAV of the same audio). The "single (music) cue / giant hallucinated blob" no longer occurs.

Revised understanding

The original failure looks far more consistent with a transient / stochastic long-form decoding collapse than with deterministic audio corruption. Temperature fallback does engage on this file (≈14% of segments retried, some up to temperature 1.0), but it does not collapse in any of my repro runs; the high-temperature retries are the plausible stochastic vector for the one-off repetition/(music) blob, but I could not reproduce it. My original two-cell "mp3 vs WAV" matrix had a sample size of one per cell and almost certainly caught a collapsed run on one side and a clean run on the other.

Net

I don't have a shareable sample that reproduces this, because I can't reproduce it at all on 0.18.0 — so there's nothing actionable here on the WhisperKit side, and I think this can be closed unless you can trigger it independently. Sorry for the noise, and thanks for the great work on WhisperKit. Happy to dig further if you ever get a hard repro.