TIKA-4745 - charset/junk/tika-eval improvements (#2861)

Author: tballison

.skills/tika-eval-compare.md

@@ -120,6 +120,16 @@ directory, plus a `summary.md` with key metrics:
 | Exception count | ≤ A | > A |
 | Total files (B) vs (A) | equal or higher | lower — missing embedded docs |
 
+### Encoding-detection evals
+
+For charset/encoding-detector changes, the summary reports don't cover it — query
+the db directly (see the **tika-eval-h2-query** skill). The detected encoding is in
+the `ENCODINGS_A`/`ENCODINGS_B` tables (`DETECTED_ENCODING`, `ENCODING_DETECTOR`,
+`DECLARED_METADATA`), **not** `PROFILES`. Key signals: per-encoding counts (e.g. CJK
+total), A→B flips by direction, and OOV on the flipped files (a flip that *worsens*
+OOV is a regression; one that *improves* it is a fix). Pair on `ID`; map back to the
+source file via `PROFILES_*.FILE_NAME` (the content hash).
+
 ### CRITICAL: Review Checklist
 
 The purpose of tika-eval is to find regressions BEFORE a release. After

.skills/tika-eval-encoding-regression.md

@@ -123,6 +123,41 @@ WHERE <enc_a/enc_b filter as above>
 ORDER BY delta ASC LIMIT 15;
 ```
 
+## Reading the signals — OOV, languageness, and FFFD together
+
+No single signal is authoritative. Use `oov` as a **secondary** signal alongside
+`languageness` (the junk-model coherence z-score) and the U+FFFD rate — each is
+right where the others are blind, so cross-check rather than ranking on any one.
+(Established 2026-06-03: a 40-file OOV-"worse" set was mostly metric artifacts
+once languageness/FFFD were brought in — only ~6 were real. But OOV is also the
+*correct* signal where languageness is blind, so neither dominates.)
+
+- **OOV can mislead** when langid shifts — a CJK/UTF-8 recovery in B is scored
+  against a different vocab → higher OOV though B is right — or when a wrong
+  decode fragments words into more short common tokens (→ higher count for the
+  WORSE decode). A common-token delta is a signal, not proof.
+- **languageness can mislead** on SBCS↔SBCS cross-script mojibake — Greek decoded
+  as KOI8-R is "coherent" Cyrillic, so `languageness` stays flat while `oov`
+  correctly flags it. Conversely languageness catches OOV's CJK/script-recovery
+  blind spot. Each covers the other's blind spot.
+- **FFFD rate** flags decode failures (illegal bytes): `num_replacement /
+  num_non_ascii` (un-diluted; `/ content_length` dilutes to ~0 on ASCII-heavy
+  docs). Tika strips C0 controls at extraction, so legal-but-wrong (C1) mojibake
+  does not surface here — that signal belongs in the detector chain, not the eval.
+- **In practice:** when the signals agree, high confidence; when they disagree
+  (OOV-worse but languageness-better, or vice versa), that file needs a look —
+  the disagreement points you at WHICH files to inspect, it does not by itself
+  declare OOV or languageness "wrong." Split OOV-worse by languageness direction
+  (query in `tika-eval-regression.adoc`).
+
+### Isolate a change against the PRIOR run, not just 3.x
+
+To see what one chain change actually did, Compare the new run against the
+*previous* 4.x run (B-new vs B-prior), not only vs 3.x. The diff should be
+*surgical* — e.g. the within-Latin letter gate moved exactly 6 files
+(IBM850 / x-MacRoman → windows-1252) vs the prior run and nothing else. A
+bigger-than-expected diff means the change fired more broadly than intended.
+
 ## Per-file detector attribution (`X-TIKA:encodingDetectionTrace`)
 
 Every JSON extract from a chain with multiple detectors carries

.skills/tika-eval-h2-query.md

@@ -40,6 +40,7 @@ it then waits on stdin and appears to hang).
 |---|---|
 | `PROFILES_A` / `PROFILES_B` | one row per extracted file: `FILE_NAME`, `MD5`, `MIME_ID`, `CONTAINER_ID`, `EMBEDDED_FILE_PATH`, `LENGTH`, `NUM_PAGES`, … (A = "before"/-a, B = "after"/-b) |
 | `CONTENTS_A` / `CONTENTS_B` | text profile per file (join on `ID`): `OOV`, `LANGUAGENESS`, `NUM_TOKENS`, `NUM_COMMON_TOKENS`, `LANG_ID_1`/`LANG_ID_PROB_1`, `TOKEN_ENTROPY_RATE`, … |
+| `ENCODINGS_A` / `ENCODINGS_B` | detected-encoding per file (join on `ID`): `DETECTED_ENCODING`, `ENCODING_DETECTOR`, `DECLARED_METADATA`. **`DETECTED_ENCODING` lives HERE, not on `PROFILES` — moved out in the encodings-table refactor; querying `PROFILES_*.DETECTED_ENCODING` now errors "Column not found".** A file with no detected encoding has no row. |
 | `CONTENT_COMPARISONS` | per-file A↔B comparison (`ID`): `DICE_COEFFICIENT`, `OVERLAP`, top token diffs |
 | `MIMES` | `MIME_ID` → `MIME_STRING` |
 | `CONTAINERS` | container id → input file path |
@@ -86,6 +87,56 @@ FROM CONTENTS_A ca JOIN CONTENTS_B cb ON ca.ID = cb.ID;
 To bring in mime/path, join `PROFILES_A pa ON pa.ID = ca.ID` (and `pb`/`cc`
 likewise on the same `id`) — all on `id`.
 
+Detected-encoding queries — `DETECTED_ENCODING` is on `ENCODINGS_A`/`ENCODINGS_B`
+(join on `ID`), NOT `PROFILES`. CJK count in B (LOWER() — `REGEXP` is case-sensitive,
+see below):
+
+```sql
+SELECT COUNT(*) FROM ENCODINGS_B
+WHERE LOWER(DETECTED_ENCODING) REGEXP 'gb|big5|euc|shift|jis|2022|949';
+```
+
+Encoding flips A→B by direction (what changed between runs):
+
+```sql
+SELECT ea.DETECTED_ENCODING a_enc, eb.DETECTED_ENCODING b_enc, COUNT(*) n
+FROM ENCODINGS_A ea JOIN ENCODINGS_B eb ON ea.ID = eb.ID
+WHERE ea.DETECTED_ENCODING <> eb.DETECTED_ENCODING
+GROUP BY a_enc, b_enc ORDER BY n DESC;
+```
+
+Map a flipped file back to its source file — `PROFILES_*.FILE_NAME` is the content
+hash (the input file is `<corpus>/<first-2-hex>/<FILE_NAME>`); join `CONTENTS` for OOV:
+
+```sql
+SELECT pb.FILE_NAME, ea.DETECTED_ENCODING a_enc, eb.DETECTED_ENCODING b_enc,
+       ca.OOV oov_a, cb.OOV oov_b
+FROM ENCODINGS_A ea JOIN ENCODINGS_B eb ON ea.ID = eb.ID
+  JOIN PROFILES_B pb ON ea.ID = pb.ID
+  JOIN CONTENTS_A ca ON ea.ID = ca.ID JOIN CONTENTS_B cb ON ea.ID = cb.ID
+WHERE LOWER(eb.DETECTED_ENCODING) REGEXP 'gb|big5|euc|shift|jis|2022|949'
+  AND NOT (LOWER(ea.DETECTED_ENCODING) REGEXP 'gb|big5|euc|shift|jis|2022|949');
+```
+
+## Gotcha: `REGEXP` is case-sensitive (silent wrong results)
+
+H2's `REGEXP` operator is **case-sensitive**, so `DETECTED_ENCODING REGEXP
+'big5|gb|euc'` does **not** match `Big5-HKSCS` or `GB18030` — and it fails
+*silently*, quietly dropping/keeping the wrong rows instead of erroring. Always
+either lowercase the column or use the inline case-insensitive flag:
+
+```sql
+-- right:
+WHERE LOWER(DETECTED_ENCODING) REGEXP 'big5|gb|euc|shift|jis|2022|949'
+-- or:
+WHERE DETECTED_ENCODING REGEXP '(?i)big5|gb|euc|shift|jis|2022|949'
+-- wrong (misses Big5-HKSCS, GB18030, Shift_JIS, ...):
+WHERE DETECTED_ENCODING REGEXP 'big5|gb|euc|shift|jis|2022|949'
+```
+
+(`DETECTED_ENCODING` is on `ENCODINGS_A`/`ENCODINGS_B` — join to `PROFILES`/`CONTENTS`
+on `ID` — populated from `X-TIKA:detectedEncoding`.)
+
 ## Tip
 
 For a quick interactive session, drop `-sql` and you get an H2 prompt; `SHOW

docs/modules/ROOT/pages/advanced/charset-detection-design.adoc

@@ -53,7 +53,7 @@ results are collected into an `EncodingDetectorContext` on the
 | `MojibusterEncodingDetector`
 | `tika-encoding-detector-mojibuster`
 | Structural UTF-32 and UTF-16 detection, UTF-8 grammar gate, HTML
-  attribute-aware stripping, then a 33-class byte-bigram NB
+  attribute-aware stripping, then a 34-class byte-bigram NB
   classifier.  STRUCTURAL for structural hits; STATISTICAL for NB
   predictions.  See <<nb-pipeline>>.
 
@@ -135,6 +135,21 @@ sequences.  Three outcomes:
 * `AMBIGUOUS` — no complete multi-byte sequence (pure ASCII, or only
   a truncated lead at probe-end).  No emission.
 
+=== ISO-2022-JP/KR/CN structural detection (pure-ASCII branch)
+
+ISO-2022 encodings are 7-bit and escape-based (`ESC $ B`, `ESC $ ) C`, …),
+so they carry no high bytes and are invisible to the byte-bigram
+classifier; without a structural check a real ISO-2022-JP page would fall
+through to the windows-1252 default and decode to gibberish.  On a
+pure-ASCII probe — the only place ISO-2022 can occur — the pipeline scans
+for the ISO-2022 designation escape and, if found, *verifies* by decoding:
+the result must contain real CJK at a near-zero replacement rate.  The
+verify rejects a stray `ESC $` in ordinary ASCII (which yields no CJK).
+On success an ISO-2022-JP/KR/CN STRUCTURAL candidate is emitted.  High-byte
+binary that happens to contain an escape sequence never reaches this
+check — it fails the pure-ASCII gate and takes the normal NB path, so it
+cannot trigger a false ISO-2022 detection.
+
 === Layer 4 — HTML stripping (content-type aware)
 
 When the probe looks like HTML/XML (explicit content-type or unknown),
@@ -155,10 +170,10 @@ just content bytes for NB feature extraction.  Optimizations:
 
 === Layer 5 — Naive Bayes byte-bigram classifier
 
-33 classes: CJK multibyte (Big5-HKSCS, EUC-JP, GB18030, Shift_JIS,
+34 classes: CJK multibyte (Big5-HKSCS, EUC-JP, GB18030, Shift_JIS,
 x-EUC-TW, x-windows-949), EBCDIC family (IBM420/424-ltr/rtl, IBM500,
 IBM1047), DOS OEM (IBM850/852/855/866), Cyrillic (KOI8-R, KOI8-U),
-Windows single-byte (1250-1258, 874), ISO-8859-3/16, Mac (x-MacRoman,
+Windows single-byte (1250-1258, 874), ISO-8859-2/3/16, Mac (x-MacRoman,
 x-mac-cyrillic), and UTF-8.
 
 Features are **stride-1 byte bigrams** — for probe bytes `b[0..N]`,
@@ -212,9 +227,10 @@ every probe length we've measured.
 
 * **Empty / near-empty probes (< 2 bytes)** → windows-1252 @ 0.1
   confidence.  WHATWG default; never returns empty result.
-* **Pure ASCII probes** (no bytes ≥ 0x80, no nulls) → windows-1252.
-  Bigram NB cannot discriminate Latin code pages on pure-ASCII
-  content; return the HTML5-canonical answer directly.
+* **Pure ASCII probes** (no bytes ≥ 0x80, no nulls) → ISO-2022 structural
+  detection first (see above); otherwise windows-1252.  Bigram NB cannot
+  discriminate Latin code pages on pure-ASCII content; return the
+  HTML5-canonical answer directly.
 * **Latin-sibling → windows-1252 rewrite** — on low-evidence probes
   (< 5 high bytes), if the top NB candidate is a non-1252 member of
   the Latin family and the probe decodes byte-identically under
@@ -223,6 +239,34 @@ every probe length we've measured.
   threshold are not emitted into the pool.  Prevents JunkFilter from
   scoring weak coincidence picks against NB's confident top.
 
+==== CJK decode-failure veto (`CjkDecodeValidator`)
+
+A legacy multi-byte CJK class (GB18030, Big5-HKSCS, Shift_JIS, EUC-JP,
+x-windows-949, x-EUC-TW) that NB picks on Latin/Cyrillic/garbage bytes is
+*false-CJK*: those bytes don't validate under the charset, so decoding
+produces many malformed/unmappable events, whereas real CJK decodes
+cleanly.  After NB, each legacy-CJK candidate is decoded under its vendor
+superset (`CharsetSupersets`) and its failure rate measured as
+`failures / high-bytes`; above ~2.5% the candidate is dropped — and if it
+was NB's only pick, the pool empties and windows-1252 wins.  Two
+corrections make the rate trustworthy:
+
+* **Decode under the vendor superset, not the strict base** — real
+  vendor-extension chars (NEC/IBM for Shift_JIS/EUC-JP, HKSCS for Big5)
+  would otherwise count as failures and penalize genuine CJK.
+* **Discount embedded UTF-8** — mixed-encoding pages (legacy CJK body +
+  UTF-8 widgets) would otherwise read as 2–9.5% failure.  The validator
+  walks the bytes and *skips* positions that begin a valid UTF-8 sequence
+  (it does NOT physically strip them — that would misalign a pure
+  legacy-CJK stream and manufacture failures), decoding the legacy charset
+  in place elsewhere.  Post-discount, real CJK (pure or mixed) is ≤1.6%
+  while genuine false-CJK stays ≥5.3%, so ~2.5% separates them.
+
+This veto catches *structurally-illegal* false-CJK only.  The
+*legal-but-wrong* class — Latin/Cyrillic bytes that form a *valid* CJK
+decode at ~0 failure — is the typicality layer's job (<<junk-filter>>),
+not this veto's.
+
 [[junk-filter]]
 == JunkFilterEncodingDetector — text-quality arbitration
 
@@ -261,6 +305,36 @@ For plain first-match-wins, omit JunkFilter (see <<opting-out-of-arbitration>>).
 . **Pairwise tournament** — first candidate seeds champion; each
   challenger compared via `JunkDetector.compare`; higher z-score wins.
 
+=== Post-tournament demote gates
+
+Two demote-only refinements run after the champion is chosen.  Each fires only
+to *demote* the champion across one boundary the whole-text z-score reads
+poorly under COMMON-dilution; neither can promote, so they cannot cost a
+confident detection.
+
+* **CJK family gate** — the whole-text z coin-flips on the CJK/non-CJK boundary
+  when markup and digits decode identically and swamp the few discriminating
+  high bytes.  A script-letter "diff" z — scored over only the `>= 0x80`
+  letters/ideographs, where candidates actually differ — reads that boundary
+  cleanly.  If the champion is CJK and the best non-CJK diff-z beats the best
+  CJK diff-z by `FAMILY_DIFF_MARGIN` (2.0), demote to the best non-CJK
+  candidate.  The reverse (promote to CJK) regressed at scale and is
+  unnecessary — genuine CJK is `<meta>`-declared upstream.
+
+* **Within-Latin letter gate** — among single-byte Latin siblings the z also
+  coin-flips, occasionally promoting a DOS-OEM / Mac charset (IBM850,
+  x-MacRoman) whose high bytes decode to box-drawing / symbols over the
+  windows-1252 truth.  Cased-letter count reads this where typicality cannot:
+  if the champion is a Latin SBCS, a windows-1252 candidate is present, the
+  probe is high-byte-dense, and windows-1252 decodes clearly more cased
+  high-byte letters (by a margin), demote to windows-1252.  Directional — a
+  genuine Central-European / DOS document has *more* letters under its true
+  charset, so the gate stays silent.  Latin-scoped, so it never crosses the
+  CJK boundary (the family gate's job) or touches a non-Latin SBCS, whose
+  Cyrillic/Greek cased letters would pollute the count.  Shares the
+  `HighByteLetterStats` letter counter with Mojibuster's Western-Latin sibling
+  fallback.
+
 === JunkDetector scoring
 
 `JunkDetector` partitions decoded text into maximal Unicode-script runs
@@ -485,7 +559,7 @@ can't encode typographic characters).
   value, vocabulary size, and each trained bigram as
   `(uint16 bigram, int8 logP)` pairs.
 
-Files for the shipped 33-class model are ~1 MB on disk.  Loader
+Files for the shipped 34-class model are ~1 MB on disk.  Loader
 materializes a dense `logP8[65 536 × numClasses]` array filled with
 per-class unseen floors, overwritten by trained pairs.  Working-set
 memory: ~2 MB.
@@ -498,7 +572,7 @@ with feature hashing.  The move to NB was driven by:
 * **Speed**: direct bigram indexing removes the hash + bucket-lookup
   cost.  Inner loop is `score[c] += logP[b × numClasses + c] × idf[b]`
   with no branching (zero-IDF bigrams are skipped before the class
-  loop).  Measured ~15 µs on a full 1 KB probe for 33 classes.
+  loop).  Measured ~15 µs on a full 1 KB probe for 34 classes.
 * **Memory layout**: bigram-major byte arrays fit in L3 cache for the
   full table.  Sequential access through the hot loop is cache-line
   efficient.

docs/modules/ROOT/pages/advanced/integration-testing/tika-eval-regression.adoc

@@ -339,7 +339,8 @@ waiting on stdin).
 
 Key tables: `profiles_a`/`profiles_b` (one row per extracted file: `file_name`,
 `mime_id`, `length`, …), `contents_a`/`contents_b` (text profile: `oov`,
-`languageness`, `num_tokens`, `lang_id_1`, …), `content_comparisons`
+`languageness`, `num_tokens`, `lang_id_1`, `num_replacement` (U+FFFD count),
+`num_non_ascii`, …), `content_comparisons`
 (`dice_coefficient`, `overlap`), `mimes`, `containers`. *A and B are paired by
 `id`* — the same row `id` is the same file in both runs (this is how the built-in
 reports join: `join profiles_b pb on pa.id = pb.id`). Always join on `id`.
@@ -351,6 +352,24 @@ SELECT SUM(CASE WHEN cb.oov < ca.oov THEN 1 ELSE 0 END) AS oov_better,
        SUM(CASE WHEN cb.oov > ca.oov THEN 1 ELSE 0 END) AS oov_worse
 FROM contents_a ca JOIN contents_b cb ON ca.id = cb.id;
 
+-- NOTE: OOV is one signal, not the verdict -- read it with languageness and the
+-- FFFD rate (use OOV as a secondary signal).  OOV can mislead (a langid shift,
+-- e.g. a CJK decode recovered in B, inflates oov_worse even when B is correct; a
+-- wrong decode that fragments words can LOWER OOV), and languageness can mislead
+-- on SBCS-cross-script mojibake -- each is right where the other is blind.  When
+-- OOV-worse and languageness disagree, that file needs a look (split below):
+SELECT SUM(CASE WHEN cb.languageness > ca.languageness + 0.2 THEN 1 ELSE 0 END) AS lang_better_oov_lied,
+       SUM(CASE WHEN cb.languageness < ca.languageness - 0.2 THEN 1 ELSE 0 END) AS lang_worse_real_candidate
+FROM contents_a ca JOIN contents_b cb ON ca.id = cb.id
+WHERE cb.oov > ca.oov + 0.02 AND ca.languageness > -90 AND cb.languageness > -90;
+
+-- FFFD decode-failure rate, un-diluted (over non-ASCII chars, NOT total length,
+-- which dilutes to ~0 on ASCII-dominated docs)
+SELECT ROUND(100.0 * cb.num_replacement / NULLIF(cb.num_non_ascii, 0), 1) AS fffd_pct,
+       cb.num_replacement, cb.num_non_ascii
+FROM contents_b cb WHERE cb.num_replacement > 0
+ORDER BY cb.num_replacement DESC FETCH FIRST 20 ROWS ONLY;
+
 -- net common-tokens A vs B (headline "more real text recovered" metric)
 SELECT SUM(ca.num_common_tokens) AS common_a,
        SUM(cb.num_common_tokens) AS common_b,

tika-core/src/main/java/org/apache/tika/detect/CharsetSupersets.java

@@ -42,6 +42,7 @@
  *   <li>GB2312 → GB18030 (GB18030 is a strict superset of both GB2312 and GBK)</li>
  *   <li>GBK → GB18030 (GB18030 is a strict superset; enables 4-byte extension sequences)</li>
  *   <li>Shift_JIS → windows-31j (MS932 is a strict superset with NEC/IBM extensions)</li>
+ *   <li>EUC-JP → x-eucJP-Open (EUC packing of the NEC/IBM vendor extensions)</li>
  * </ul>
  */
 public final class CharsetSupersets {
@@ -59,6 +60,7 @@ public final class CharsetSupersets {
         m.put("GB2312",    "GB18030");
         m.put("GBK",       "GB18030");
         m.put("Shift_JIS", "windows-31j");
+        m.put("EUC-JP",    "x-eucJP-Open");
         SUPERSET_MAP = Collections.unmodifiableMap(m);
     }