TIKA-4745-follow-on-junk-improvements

[tballison]

Thanks for your contribution to Apache Tika! Your help is appreciated!

Before opening the pull request, please verify that

• there is an open issue on the Tika issue tracker which describes the problem or the improvement. We cannot accept pull requests without an issue because the change wouldn't be listed in the release notes.
• the issue ID (TIKA-XXXX)
  • is referenced in the title of the pull request
  • and placed in front of your commit messages surrounded by square brackets ([TIKA-XXXX] Issue or pull request title)
• commits are squashed into a single one (or few commits for larger changes)
• Tika is successfully built and unit tests pass by running ./mvnw clean test
• there should be no conflicts when merging the pull request branch into the recent main branch. If there are conflicts, please try to rebase the pull request branch on top of a freshly pulled main branch
• if you add new module that downstream users will depend upon add it to relevant group in tika-bom/pom.xml.

We will be able to faster integrate your pull request if these conditions are met. If you have any questions how to fix your problem or about using Tika in general, please sign up for the Tika mailing list. Thanks!

[Copilot]

Pull request overview

This PR follows up on TIKA-4745 by refining “junk detection” behavior (training + inference) to better handle decode corruption signals (especially U+FFFD), case-folding for ALL-CAPS text, and script/run tokenization so training and inference use the same bigram emission logic. It also extends the HTML parser to record the actual decoding charset (when a superset is used) via TikaCoreProperties.DECODED_CHARSET.

Changes:

• HTML: Record decodedCharset metadata when decoding uses a superset charset (e.g., EUC-KR → x-windows-949) and add a regression test.
• Junkdetect: Switch z1 bigram enumeration to run/token based emission (with sentinels), add case-folded backoff for all-caps, exclude anomalies from z1 while strengthening z6 usage, and align normalization to NFKC across training/inference.
• Mojibuster / encoding detection: Consolidate charset superset decoding via CharsetSupersets.decodeAs(...) and remove now-unused classes in the mojibuster module.

Reviewed changes

Copilot reviewed 12 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tika-parsers/.../HtmlParserTest.java	Adds a regression test asserting EUC-KR detection while decoding via MS949 superset and recording decodedCharset.
tika-parsers/.../JSoupParser.java	Sets TikaCoreProperties.DECODED_CHARSET when decode charset differs from detected charset.
tika-ml/.../JunkDetectorRoundTripTest.java	Adds tokenizer/run emission tests and validates case-folded backoff behavior.
tika-ml/.../DecodeCorruptionDiscriminationTest.java	New monotonicity/regression tests ensuring clean text outscores decode-corrupted variants.
tika-ml/.../tools/TrainJunkModel.java	Aligns training tokenization with inference, floors degenerate z1 sigma, switches normalization to NFKC, and adds new contrastive training pairs for FFFD/case.
tika-ml/.../tools/BuildJunkTrainingData.java	Switches normalization to NFKC to match inference and improve compatibility-form handling (e.g., half-width katakana).
tika-ml/.../TextQualityFeatures.java	Refines replacement ratio denominator and excludes U+FFFD from script density computation.
tika-ml/.../JunkFilterEncodingDetector.java	Uses CharsetSupersets.decodeAs(...) for vendor-superset decoding.
tika-ml/.../JunkDetector.java	Major updates: NFKC normalization, fallback logit includes z6, case-folded bigram scoring, sentinel-based run bigram emission, and new boundary/glue logic.
tika-encoding-detectors/.../CjkDecodeValidator.java	Uses CharsetSupersets.decodeAs(...) for superset decoding during validation.
tika-encoding-detectors/.../ScoredCandidate.java	Removed (no longer used).
tika-encoding-detectors/.../CosineFamilyArbiter.java	Removed (no longer used).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 14 changed files in this pull request and generated 1 comment.