Skip to content

fix(transcription): shared cross-provider hallucination filtering (follow-up to #679) - #747

Open
Hus-Mek wants to merge 2 commits into
Zackriya-Solutions:devtestfrom
Hus-Mek:fix/cross-provider-text-cleanup
Open

fix(transcription): shared cross-provider hallucination filtering (follow-up to #679)#747
Hus-Mek wants to merge 2 commits into
Zackriya-Solutions:devtestfrom
Hus-Mek:fix/cross-provider-text-cleanup

Conversation

@Hus-Mek

@Hus-Mek Hus-Mek commented Aug 30, 2026

Copy link
Copy Markdown

Follow-up to #679, split out per review so the VAD fix and this transcript-output policy change are reviewed independently.

What this fixes

Hallucination filtering only ran on local Whisper, and only knew English.

clean_repetitive_text and is_meaningless_output were private associated functions on WhisperEngine. The Parakeet arm and the provider (cloud) arm in worker.rs both stored result.text.trim() verbatim, so they got no filtering at all. The denylist was also entirely English ("thank you for watching", "like and subscribe", "applause").

Net effect: an English meeting got cleaned and a non-English meeting did not. A real 26-minute Arabic recording stored these unfiltered:

stored text what it is
اشتركوا في القناة "subscribe to the channel"
ترجمة نانسي قنقر a subtitler credit, on a 0.15 s clip
شكرا x14 standalone - the Arabic analogue of Whisper's English "Thank you." attractor

What changed

  • new audio/transcription/text_cleanup.rs - the cleanup routines extracted into one module and called from all three engine arms (local Whisper, Parakeet, remote providers), with Arabic patterns added alongside the English ones and grouped by language so further languages are additive.
  • whisper_engine.rs - the now-shared code removed; delegates to the new module. (This is why the diff is move-heavy: ~155 lines relocated out of here.)
  • worker.rs - the Parakeet and provider arms now call text_cleanup::clean_transcript_text instead of storing raw text. No change to the confidence gate - the 0.3 threshold removed by Fix: stop dropping transcripts below confidence threshold (#675) #681 is not touched or reintroduced here.
  • transcription/mod.rs - registers the new module.

Behaviour, and the #675 correction

An earlier version of this work (in the original #679) included a standalone-filler denylist that dropped "thank you", "okay", "thanks", "you", "bye" and Arabic شكرا when they were a whole segment. That was reverted before this split: nothing in the text distinguishes شكرا decoded from 300 ms of silence from شكرا a participant actually said, so the list traded a cosmetic problem for a semantic one - and #675 is that exact failure already reported. Short-clip hallucination belongs in the provider layer gating on Whisper's own avg_logprob / compression_ratio, not in a text denylist.

What this PR filters by text is only unambiguous multi-word boilerplate (اشتركوا في القناة, subtitler credits, thanks for watching, like and subscribe) - phrases no meeting contains, so there is no genuine reading to lose.

Degenerate repetition is still caught language-agnostically: the repetition ratio is measured on the original text rather than after collapsing (measuring after collapsing hid it - a collapsed run leaves one word, ratio zero). شكرا شكرا شكرا شكرا شكرا is still discarded without the code needing to know what the word means.

Verification

$ cargo test --lib text_cleanup

running 14 tests
test audio::transcription::text_cleanup::tests::drops_arabic_subscribe_boilerplate ... ok
test audio::transcription::text_cleanup::tests::drops_arabic_subtitler_credit ... ok
test audio::transcription::text_cleanup::tests::drops_english_boilerplate_as_before ... ok
test audio::transcription::text_cleanup::tests::empty_and_whitespace_are_empty ... ok
test audio::transcription::text_cleanup::tests::discards_fully_degenerate_repetition ... ok
test audio::transcription::text_cleanup::tests::keeps_arabic_thank_you_inside_a_real_sentence ... ok
test audio::transcription::text_cleanup::tests::collapses_repeated_phrases ... ok
test audio::transcription::text_cleanup::tests::collapses_repeated_words ... ok
test audio::transcription::text_cleanup::tests::keeps_code_switched_arabic_english ... ok
test audio::transcription::text_cleanup::tests::keeps_ordinary_arabic_speech ... ok
test audio::transcription::text_cleanup::tests::keeps_short_acknowledgements_issue_675 ... ok
test audio::transcription::text_cleanup::tests::keeps_short_genuine_replies ... ok
test audio::transcription::text_cleanup::tests::keeps_tatweel_stretched_words ... ok
test audio::transcription::text_cleanup::tests::single_character_runs_are_junk ... ok

test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 212 filtered out

keeps_short_acknowledgements_issue_675 asserts that OK, Yes, No, Sure, Got it, Thanks, Okay., Bye plus شكرا, شكراً, شكرا لك, نعم, طيب, لا and a tatweel-stretched شكـــرا all survive cleaning unchanged - i.e. this PR does not make #675 worse from a second direction.

The 14 tests cover both directions deliberately: the artifacts that must be removed, and ordinary Arabic, code-switched Arabic/English, and short genuine replies that must survive.

Note on diff size

The diff is ~506 changed lines, but ~155 of the deletions are code relocated verbatim out of whisper_engine.rs into the new module; net new logic is small. Happy to split further if you'd prefer the extraction and the cross-provider wiring in separate commits, but they don't stand alone cleanly.

…ocal Whisper

`clean_repetitive_text` and `is_meaningless_output` were private associated
functions on `WhisperEngine`, reachable only from the local-Whisper path. The
Parakeet arm and the trait-based provider arm in `worker.rs` both stored
`result.text.trim()` verbatim, so Whisper's subtitle-corpus artifacts were
stripped on one engine and persisted on the others.

The denylist was also entirely English ("thank you for watching", "like and
subscribe", "applause"). Whisper emits the same class of artifact in whichever
language it decided it was hearing, so the filter silently applied to English
meetings and to nothing else. A real Arabic meeting stored these unfiltered:

  اشتركوا في القناة    "subscribe to the channel"
  ترجمة نانسي قنقر     a subtitler credit
  شكرا                 14 standalone occurrences, the Arabic analogue of
                       Whisper's English "Thank you." attractor on near-silent input

Extracts all five routines into `audio/transcription/text_cleanup.rs` and calls
it from all three engine arms. Adds the Arabic patterns alongside the English
ones, grouped by language so further languages are additive.

Behaviour changes beyond relocation:

- Filler such as "شكرا" is matched against the whole trimmed segment rather than
  as a substring, so it is dropped when it IS the segment but kept inside a real
  sentence. Trailing punctuation and Arabic tatweel are normalised first so
  orthographic variants collapse to one form.
- `is_meaningless_output` is re-checked after repetition collapsing, because
  "شكرا شكرا شكرا" only becomes recognisable as standalone filler once the
  repetition has been removed. Caught by a test.
- `remove_word_repetitions` had identical `if`/`else` bodies (`repeat_count` is 1
  in the else branch); collapsed to the single shared path.

14 unit tests cover both directions: the artifacts that must be removed, and
ordinary Arabic, code-switched Arabic/English and short genuine replies that must
survive.
…sue Zackriya-Solutions#675)

The previous commit added a standalone-filler denylist ("thank you", "okay",
"thanks", Arabic "شكرا" and variants) that dropped those words when they made up
a whole segment. That was wrong, and it would have compounded a known bug.

Whisper genuinely does hallucinate exactly those words on near-silent input, which
is what made the list tempting. But they are also completely ordinary meeting
speech, and NOTHING IN THE TEXT DISTINGUISHES THE TWO — "شكرا" decoded from 300ms
of silence and "شكرا" decoded from someone saying it are byte-identical.

So the denylist traded a cosmetic problem (a stray "thanks" in the transcript) for
a semantic one (a participant's answer silently missing). Issue Zackriya-Solutions#675 is precisely
that failure reported for "OK" / "Yes" / "Thanks" via a different mechanism, so
adding a text-only filler list would have made a filed bug worse from a second
direction.

Short-clip hallucination belongs where the evidence actually is: the provider
layer, gating on Whisper's own `avg_logprob` / `compression_ratio`. Only
unambiguous multi-word boilerplate — "subscribe to the channel", subtitler
credits, "thanks for watching" — is filtered by text, because no meeting contains
those phrases.

Degenerate repetition is still caught, now language-agnostically: the repetition
ratio is measured on the ORIGINAL text rather than after collapsing. Collapsing
first hid it, because a collapsed run leaves one word and one word has a
repetition ratio of zero. "شكرا شكرا شكرا شكرا شكرا" is still discarded, without
the code needing to know what the word means.

Replaces the two inverted tests with `keeps_short_acknowledgements_issue_675`,
covering OK / Yes / No / Sure / Got it / Thanks / Bye and the Arabic and
tatweel-stretched equivalents. 14 tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant