TIKA-4745 -- efficiency improvements

[tballison]

Thanks for your contribution to Apache Tika! Your help is appreciated!

Before opening the pull request, please verify that

• there is an open issue on the Tika issue tracker which describes the problem or the improvement. We cannot accept pull requests without an issue because the change wouldn't be listed in the release notes.
• the issue ID (TIKA-XXXX)
  • is referenced in the title of the pull request
  • and placed in front of your commit messages surrounded by square brackets ([TIKA-XXXX] Issue or pull request title)
• commits are squashed into a single one (or few commits for larger changes)
• Tika is successfully built and unit tests pass by running ./mvnw clean test
• there should be no conflicts when merging the pull request branch into the recent main branch. If there are conflicts, please try to rebase the pull request branch on top of a freshly pulled main branch
• if you add new module that downstream users will depend upon add it to relevant group in tika-bom/pom.xml.

We will be able to faster integrate your pull request if these conditions are met. If you have any questions how to fix your problem or about using Tika in general, please sign up for the Tika mailing list. Thanks!

[Copilot]

Pull request overview

This PR (TIKA-4745) focuses on runtime and memory efficiency improvements across Tika’s ML-based junk/encoding detection and some text preprocessing paths. It primarily reduces repeated probe reads/processing, shrinks serialized model representations, and avoids expensive work when it can be proven unnecessary.

Changes:

• Replaced JunkDetector bigram open-addressing tables with sorted-occupied arrays (binary-search lookup) and introduced delta+LEB128 serialization for bigram keys.
• Added a per-detection EncodingProbeCache and wired it into detectors to prevent repeated probe reads/tag-stripping across detector chains.
• Reduced unnecessary allocations/scans in probe reading (AdaptiveProbe growth-on-demand), HTML meta sniffing, and CharSoup preprocessing (marker-gated regex).

Reviewed changes

Copilot reviewed 13 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/JunkDetectorRoundTripTest.java	Updates tests to build sorted-occupied bigram tables directly (no OA insertion).
tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/tools/TrainJunkModel.java	Builds sorted key/value bigram entries instead of OA hash tables.
tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java	Uses EncodingProbeCache to avoid rereading the same probe within a detection.
tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkDetector.java	Switches bigram lookup to binary search and removes OA mixing.
tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/BigramTables.java	Updates bigram table docs + adds delta/LEB128 key serialization.
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/TrainNaiveBayesBigram.java	Writes NB bigram model keys as sorted varint deltas to reduce size.
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/NaiveBayesBigramEncodingDetector.java	Reads NB bigram model keys as varint deltas.
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java	Uses EncodingProbeCache when available to reuse the probe.
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/AdaptiveProbe.java	Avoids eager 2× rawCap allocations by growing buffers on demand.
tika-encoding-detectors/tika-encoding-detector-html/src/main/java/org/apache/tika/parser/html/HtmlEncodingDetector.java	Adds a byte-level <meta precheck to skip expensive decode/regex when impossible to match.
tika-langdetect/tika-langdetect-charsoup-core/src/main/java/org/apache/tika/langdetect/charsoup/CharSoupFeatureExtractor.java	Skips URL/email regex scans unless marker substrings are present.
tika-core/src/main/java/org/apache/tika/detect/EncodingProbeCache.java	New per-detection probe cache object.
tika-core/src/main/java/org/apache/tika/detect/EncodingDetectorContext.java	Adds and exposes a per-detection EncodingProbeCache.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 20 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/AdaptiveProbe.java:79

• IOUtils.read(...) can return -1 at EOF. The current code adds this to total, which can make total negative on an empty stream and lead to incorrect stripping logic or NegativeArraySizeException on return. Handle -1 explicitly before updating total.

                int n = IOUtils.read(tis, buf, total, want);
                total += n;
                HtmlByteStripper.Result r =
                        HtmlByteStripper.stripTags(buf, 0, total, stripDst, 0);
                int content = r.tagCount > 0 ? r.length : total;
                if (content >= contentTarget || n < want) {
                    break; // enough body text, or EOF
                }

[Copilot]

Pull request overview

Copilot reviewed 15 out of 20 changed files in this pull request and generated 3 comments.