Skip to content

Fix UTF-8 false-positive on short probes with tolerated errors (regression from #2882 + #2878) - #2888

Merged
tballison merged 2 commits into
mainfrom
fix-merge-conflict
Jun 10, 2026
Merged

Fix UTF-8 false-positive on short probes with tolerated errors (regression from #2882 + #2878)#2888
tballison merged 2 commits into
mainfrom
fix-merge-conflict

Conversation

@tballison

@tballison tballison commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a CI regression introduced by the interaction of #2878 and #2882:

  • TIKA-4745 -- efficiency improvements #2878 added a utf8Tolerated path: when a probe is NOT_UTF8 but has ≤ UTF8_MAX_TOLERATED_ERRORS error events, the UTF-8 NB result is kept (not disqualified) and a STRUCTURAL UTF-8 candidate is also emitted.
  • make mojibuster default #2882 changed the return strategy: when the top result is STRUCTURAL, return only that one result and skip JunkFilter arbitration.
  • Combined effect: A short probe like "the name is \u00e1ndre" in ISO-8859-1 (17 bytes, 1 UTF-8 error event) was being promoted to STRUCTURAL UTF-8, which then short-circuited JunkFilter, preventing it from comparing UTF-8 vs windows-1252. Result: UTF-8 returned instead of windows-1252.

Fix: Only emit the STRUCTURAL UTF-8 candidate for LIKELY_UTF8 (definitively clean grammar). For utf8Tolerated (NOT_UTF8 but within tolerance), NB's UTF-8 result is already kept as a STATISTICAL candidate — JunkFilter has enough signal to arbitrate correctly between UTF-8 and windows-1252.

Failing tests fixed: TXTParserTest.testUseIncomingCharsetAsHint and TXTParserTest.testUsingCharsetInContentTypeHeader.

@tballison tballison changed the title merge conflict Fix UTF-8 false-positive on short probes with tolerated errors (regression from #2882 + #2878) Jun 10, 2026
@tballison
tballison merged commit 360b3d3 into main Jun 10, 2026
5 checks passed
@tballison
tballison deleted the fix-merge-conflict branch September 3, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant