TIKA-4720 -- Move charset detection to byte-bigram Naive Bayes pipeline (#2784)

* Move charset detection to byte-bigram Naive Bayes pipeline

Author: tballison

docs/modules/ROOT/pages/advanced/charset-detection-design.adoc

@@ -18,119 +18,113 @@
 = Configuring Encoding Detectors
 
 Tika uses a chain of _encoding detectors_ to determine the character encoding
-of plain text and HTML content.  The chain is controlled by
-`DefaultEncodingDetector`, which loads detectors via the Java service-provider
-interface (SPI) and runs them in registration order.
+of plain text and HTML content.  `DefaultEncodingDetector` loads detectors
+via the Java service-provider interface (SPI) and runs them in registration
+order.  See xref:../advanced/charset-detection-design.adoc[Charset Detection Pipeline]
+for design details.
 
 == Default Detection Chain
 
-The default chain when `tika-charset-detectors-core` is on the classpath:
+With the stock dependencies on the classpath:
 
-[cols="1,2,1"]
+[cols="1,2,2"]
 |===
 |Step |Detector |Returns non-null when…
 
 |1
-|`http-header-encoding-detector`
-|A `charset=` parameter is present in the `Content-Type` metadata field
-(e.g. populated from an HTTP response header).
+|`bom-detector`
+|A UTF-8, UTF-16 LE/BE, or UTF-32 LE/BE byte-order mark is present.
+Emits DECLARATIVE.
 
 |2
-|`bom-encoding-detector`
-|A UTF-8, UTF-16 LE/BE, or UTF-32 LE/BE byte-order mark is present.
+|`metadata-charset-detector`
+|A `charset=` parameter is present in the `Content-Type` metadata field
+(populated from an HTTP response header or similar).  Emits DECLARATIVE.
 
 |3
-|`html-encoding-detector`
-|An HTML `<meta charset="…">` or `Content-Type` http-equiv tag is found
-(fast lenient regex matcher, curated WHATWG label aliases).
+|`naive-bayes-pipeline-encoding-detector`
+|A structural UTF-32 check, structural UTF-16 specialist, UTF-8 grammar
+gate, and 33-class byte-bigram Naive Bayes classifier.  STRUCTURAL for
+structural hits; STATISTICAL for NB predictions.
 
 |4
-|`ml-encoding-detector`
-|The built-in statistical model classifies the byte stream (~46 encodings,
-~185 KB model bundled as a resource).
-
-|5 _(if present)_
-|`universal-encoding-detector`
-|State-machine structural prober (juniversalchardet fork).  Automatically
-joins the chain when `tika-charset-detectors-universal` is on the classpath.
-Complements ML: excels at short or repetitive CJK byte sequences (ZIP entry
-names, single-word filenames) where statistical models lack sufficient texture.
+|`html-encoding-detector`
+|An HTML `<meta charset="…">` or `<meta http-equiv="Content-Type">` tag
+is found (fast lenient regex matcher, curated WHATWG label aliases).
+Emits DECLARATIVE.
 
-|6 _(if present)_
+|5
 |`charsoup-encoding-detector`
-|A `MetaEncodingDetector` that runs after all base detectors.  When they all
-agree it returns the unanimous result; when they disagree it uses language-
-detection scoring — with a junk-ratio fallback (fewest undefined codepoints
-wins) for content too short for reliable language detection.
+|`MetaEncodingDetector` — arbitrates across all base-detector candidates
+using a character-bigram language model.  When one or more base detectors
+return a result, CharSoup decodes the probe with each candidate charset
+and picks the one whose decoded text scores best for any supported
+language.  Always runs last.
 |===
 
-NOTE: `universal-encoding-detector` and `charsoup-encoding-detector` are
-supplied by separate optional modules (`tika-charset-detectors-universal` and
-`tika-langdetect-charsoup` respectively).  Each is loaded automatically via
-SPI when its module is on the classpath and requires no extra configuration.
-
-== Design Rationale
-
-The chain combines two complementary detection strategies:
-
-* **Statistical (ML)** — learns byte-bigram distributions from training data.
-  Works well for documents with enough varied content (~100+ bytes).
-* **Structural (Universal)** — applies encoding-spec constraints (is this a
-  valid lead+trail byte pair for Shift_JIS / EUC-JP / Big5 / GBK?).  Works
-  on as few as two bytes and is unaffected by content length.
-
-Rules beat statistics at the extremes (very short or highly structured input);
-statistics beat rules in the ambiguous middle where distributions are rich.
-`charsoup-encoding-detector` arbitrates when they disagree.
+NOTE: `charsoup-encoding-detector` is supplied by
+`tika-encoding-detector-charsoup` and loads automatically via SPI when
+the module jar is present.  Omit it to get strict first-match-wins
+behaviour that honours declarations authoritatively — see
+<<opting-out-of-charsoup>>.
 
 == Available Detectors
 
 All detectors implement `org.apache.tika.detect.EncodingDetector` and can be
-referenced by name in JSON configuration.
+referenced by their SPI name in JSON configuration.
 
 [cols="2,2,3"]
 |===
 |Name |Module |Description
 
-|`http-header-encoding-detector`
-|`tika-charset-detectors-core`
-|Reads `charset=` from the `Content-Type` metadata field.  In the default chain.
+|`bom-detector`
+|`tika-core`
+|Reads the first 4 bytes for BOM signatures.  In the default chain.
+
+|`metadata-charset-detector`
+|`tika-core`
+|Reads declarative hints (`Content-Type` charset, `Content-Encoding`) from
+the `Metadata` object.  Applies WHATWG label normalization
+(ISO-8859-1 and US-ASCII → windows-1252).  In the default chain.
 
-|`bom-encoding-detector`
-|`tika-charset-detectors-core`
-|Byte-order mark detection (UTF-8/16/32).  In the default chain.
+|`naive-bayes-pipeline-encoding-detector`
+|`tika-encoding-detector-mojibuster`
+|Byte-bigram Naive Bayes classifier plus structural detectors for UTF-32
+and UTF-16 and a UTF-8 grammar gate.  33 classes including CJK
+multi-byte, EBCDIC variants, DOS code pages, Cyrillic, Windows single-byte,
+ISO-8859-3/16, Mac, and UTF-8.  In the default chain.  See
+xref:../advanced/charset-detection-design.adoc[the design doc].
 
 |`html-encoding-detector`
-|`tika-charset-detectors-core`
+|`tika-encoding-detector-html`
 |Fast lenient regex matcher for `<meta charset>` / `http-equiv` tags, with a
 curated subset of WHATWG label aliases.  In the default chain.
 
-|`ml-encoding-detector`
-|`tika-charset-detectors-core`
-|Statistical multinomial logistic regression model (~46 encodings).
-In the default chain.
-
-|`universal-encoding-detector`
-|`tika-charset-detectors-universal`
-|State-machine structural prober; wraps the `com.github.albfernandez:juniversalchardet`
-fork.  Auto-registers when the module jar is on the classpath.
-
 |`standard-html-encoding-detector`
-|`tika-charset-detectors-core`
+|`tika-encoding-detector-html`
 |Spec-strict WHATWG prescan algorithm.  Not in the default chain — opt in
 explicitly if you need strict WHATWG tokenisation (e.g. ignoring charset
 declarations inside comments or other contexts the lenient regex may match).
 
+|`charsoup-encoding-detector`
+|`tika-encoding-detector-charsoup`
+|Language-aware arbitrator (`MetaEncodingDetector`).  In the default
+chain when the module jar is on the classpath; always runs last.
+
+|`mojibuster-encoding-detector`
+|`tika-encoding-detector-mojibuster`
+|Legacy maxent charset classifier.  Still available for explicit opt-in;
+not auto-registered.
+
 |`icu4j-encoding-detector`
-|`tika-charset-detectors-icu4j`
-|Wraps ICU4J `CharsetDetector`.  _Legacy_ — the ML + Universal chain
-supersedes it for most use cases.  Available for explicit opt-in when
-`com.ibm.icu:icu4j` is already on the classpath.
+|`tika-encoding-detector-icu4j`
+|Wraps ICU4J `CharsetDetector`.  Legacy — the NB pipeline supersedes it
+for most cases.  Not auto-registered.
 
-|`charsoup-encoding-detector`
-|`tika-langdetect-charsoup`
-|Language-aware arbitrator (`MetaEncodingDetector`).  Auto-registers when
-the module jar is on the classpath; always runs last.
+|`universal-encoding-detector`
+|`tika-encoding-detector-universal`
+|State-machine structural prober (juniversalchardet fork).  Not
+auto-registered; opt in if you specifically need it.
 |===
 
 == Configuration Examples
@@ -143,63 +137,76 @@ the module jar is on the classpath; always runs last.
   "encoding-detectors": [
     {
       "default-encoding-detector": {
-        "exclude": ["bom-encoding-detector"]
+        "exclude": ["bom-detector"]
       }
     }
   ]
 }
 ----
 
-=== Restrict to a lightweight chain (no Universal, no CharSoup)
+[[opting-out-of-charsoup]]
+=== Strict declared-charset honoring (no CharSoup arbitration)
 
-Useful in resource-constrained environments when you only need the core
-statistical chain:
+By default, CharSoup arbitrates across all candidates including DECLARATIVE
+ones — which means a `<meta charset>` tag is not automatically
+authoritative.  If you want declarations to win unconditionally (HTML5-spec
+behaviour), omit CharSoup:
 
 [source,json]
 ----
 {
   "encoding-detectors": [
-    {"http-header-encoding-detector": {}},
-    {"bom-encoding-detector": {}},
+    {"bom-detector": {}},
+    {"metadata-charset-detector": {}},
     {"html-encoding-detector": {}},
-    {"ml-encoding-detector": {}}
+    {"naive-bayes-pipeline-encoding-detector": {}}
   ]
 }
 ----
 
+Without CharSoup, `CompositeEncodingDetector` runs in first-match-wins
+mode.  Trade-off: lying declarations (common on the legacy web)
+propagate unfiltered.  See
+xref:../advanced/charset-detection-design.adoc#opting-out-of-arbitration[the design doc]
+for details.
+
 === Configure the HTML detector's read limit
 
-The default limit is 8 192 bytes.  Raise it if your HTML documents embed
-large `<script>` blocks before the `<meta charset>` declaration.
+`html-encoding-detector` reads up to 65 536 bytes by default when
+scanning for the `<meta charset>` tag.  Raise it if your documents embed
+large `<script>` blocks before the meta tag (TIKA-2485):
 
 [source,json]
 ----
 {
   "encoding-detectors": [
-    {"http-header-encoding-detector": {}},
-    {"bom-encoding-detector": {}},
+    {"bom-detector": {}},
+    {"metadata-charset-detector": {}},
     {
       "html-encoding-detector": {
-        "markLimit": 65536
+        "markLimit": 131072
       }
     },
-    {"ml-encoding-detector": {}}
+    {"naive-bayes-pipeline-encoding-detector": {}},
+    {"charsoup-encoding-detector": {}}
   ]
 }
 ----
 
-=== Recreate the pre-4.x default (HTML + juniversalchardet + ICU4J)
+=== Legacy chain (ICU4J + juniversalchardet)
 
-Not recommended — the new chain is strictly better — but possible for
-regression testing or comparison:
+Not recommended — the NB pipeline is strictly better on accuracy and
+latency — but available for regression testing or comparison:
 
 [source,json]
 ----
 {
   "encoding-detectors": [
+    {"bom-detector": {}},
+    {"metadata-charset-detector": {}},
     {"html-encoding-detector": {}},
-    {"universal-encoding-detector": {}},
-    {"icu4j-encoding-detector": {}}
+    {"icu4j-encoding-detector": {}},
+    {"universal-encoding-detector": {}}
   ]
 }
 ----

docs/modules/ROOT/pages/configuration/encoding-detectors.adoc

@@ -38,7 +38,7 @@
  *
  * <p>SPI-loaded first in the default encoding-detector chain so that BOM evidence
  * reaches {@code CharSoupEncodingDetector} before any statistical detector runs.
- * {@code MojibusterEncodingDetector} strips the BOM from its own probe independently
+ * {@code NaiveBayesPipelineEncodingDetector} strips the BOM from its own probe independently
  * to ensure consistent model inference (BOMs are excluded from training data).</p>
  *
  * @since Apache Tika 0.x (moved to org.apache.tika.detect in 4.0)

tika-core/src/main/java/org/apache/tika/detect/BOMDetector.java

@@ -17,6 +17,6 @@
 # It must be loaded last; DefaultEncodingDetector's class-name sort puts
 # org.apache.tika.parser.* after org.apache.tika.ml.* after
 # org.apache.tika.detect.*, which guarantees the correct order:
-#   MojibusterEncodingDetector →
+#   NaiveBayesPipelineEncodingDetector →
 #   StandardHtmlEncodingDetector → CharSoupEncodingDetector
 org.apache.tika.langdetect.charsoup.CharSoupEncodingDetector

tika-encoding-detectors/tika-encoding-detector-charsoup/src/main/resources/META-INF/services/org.apache.tika.detect.EncodingDetector

@@ -0,0 +1,50 @@
+SLF4J(W): No SLF4J providers were found.
+SLF4J(W): Defaulting to no-operation (NOP) logger implementation
+SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
+
+=== Probe length: 20B ===
+                            N  | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
+Charset                        | ICU4J R%   S%  T3%  D%   A%  | juniv R%   S%  T3%  D%   A%  | NB   R%   S%  T3%  D%   A%  |
+----------------------------------------------------------------------------------------------------------------------------
+Big5-HKSCS              30073  |   0.0  13.9  71.6  16.4  16.6  |   0.0  44.2  44.2  46.6  46.9  |  61.1  61.1  61.2  65.1  65.1  |
+EUC-JP                  37047  |   0.0   0.0  14.7   6.6   7.6  |  64.1  64.1  64.1  71.1  72.1  |  75.9  75.9  81.9  83.2  83.9  |
+GB18030                 36859  |   0.2   0.2   7.8   6.0   6.7  |  45.8  45.8  45.8  51.7  52.5  |  77.5  77.5  78.0  83.5  83.7  |
+IBM1047                 31450  |   0.0  92.0  98.3  90.9  91.5  |   0.0   0.0   0.0   0.0   0.0  |  19.5  71.4  71.8  71.0  71.1  |
+IBM420-ltr              36865  |   0.0  83.6  91.1   0.0   0.0  |   0.0   0.0   0.0   0.0   0.0  |  67.7  70.5  70.9  67.7  67.7  |
+IBM420-rtl              37022  |   0.0  88.2  93.6   0.0   0.0  |   0.0   0.0   0.0   0.0   0.0  |  70.9  73.0  73.4  70.9  70.9  |
+IBM424-ltr              35011  |   0.0  74.6  85.3   0.0   0.0  |   0.0   0.0   0.0   0.0   0.0  |  13.4  14.4  15.5  13.4  13.4  |
+IBM424-rtl              33092  |   0.0  78.8  87.3   0.0   0.0  |   0.0   0.0   0.0   0.0   0.0  |  13.5  14.6  15.5  13.5  13.5  |
+IBM500                  31419  |  92.0  92.0  98.2  92.0  92.0  |   0.0   0.0   0.0   0.0   0.0  |  51.9  71.1  71.5  70.9  71.1  |
+IBM850                  30465  |   0.0   0.0   0.0  57.1  57.9  |   0.0   0.0   0.0  58.0  58.9  |   5.8   5.8   7.5  64.2  64.2  |
+IBM852                  35340  |   0.0   0.0   0.0  39.8  39.9  |   0.0   0.0   0.0  41.6  41.9  |  11.8  11.8  13.6  53.5  53.6  |
+IBM855                  36704  |   0.0   0.0   0.0   1.7   1.7  |  93.2  93.2  93.2  94.9  95.0  |  71.0  71.0  72.9  72.8  72.8  |
+IBM866                  36979  |  52.3  52.3  79.4  54.5  54.5  |  94.6  94.6  94.6  96.9  96.9  |  29.3  29.3  29.8  31.6  31.6  |
+ISO-8859-16             32896  |   0.0   0.0   0.0  84.2  84.6  |   0.0   0.0   0.0  82.3  82.9  |   4.2   4.2   5.8  61.6  61.6  |
+ISO-8859-3              35637  |   0.0   0.0   0.0  50.9  50.9  |   0.0   0.0   0.0  53.3  53.3  |   9.5   9.5  10.2  61.8  61.8  |
+KOI8-R                  36870  |  66.8  66.8  77.9  69.0  69.0  |  95.9  95.9  95.9  98.2  98.2  |  52.9  55.7  80.2  58.0  58.0  |
+KOI8-U                  36845  |   0.0  59.5  73.6  19.1  19.2  |   0.0  97.1  97.1  33.0  33.1  |  59.3  64.5  82.7  65.7  65.7  |
+Shift_JIS               36883  |   0.0   0.0   2.7   7.0   7.6  |  66.9  66.9  66.9  74.2  74.8  |  39.4  39.4  39.5  46.7  46.7  |
+UTF-16-BE               36789  |  36.0  36.0  38.3  36.0  36.0  |   0.0   0.0   0.0   0.0   0.0  |  94.9  95.0  95.0  94.9  94.9  |
+UTF-16-LE               36730  |  36.8  36.8  39.0  36.8  36.8  |   0.0   0.0   0.0   0.0   0.0  |  94.8  94.9  94.9  94.8  94.8  |
+UTF-32-BE               36751  | 100.0 100.0 100.0 100.0 100.0  |   0.0   0.0   0.0   0.0   0.0  |   0.0   0.0  23.7   0.0   0.0  |
+UTF-32-LE               37012  | 100.0 100.0 100.0 100.0 100.0  |   0.0   0.0   0.0   0.0   0.0  |   0.0   0.0  23.1   0.0   0.0  |
+UTF-8                   36283  |  82.3  82.3  91.8  97.7  97.7  |  82.2  82.2  82.2  98.9  98.9  |  82.6  82.6  82.6  99.3  99.3  |
+windows-1250            34472  |  11.1  53.5  88.8  83.8  83.9  |   0.0   0.0   0.0  58.8  62.5  |   6.9   6.9   9.9  53.2  53.4  |
+windows-1251            36856  |  60.4  60.6  76.1  62.2  62.2  |  74.6  74.7  74.7  76.5  76.6  |  58.8  58.8  65.7  60.6  60.6  |
+windows-1252            25636  |   3.8  65.2  94.7  88.2  88.2  |   0.0  98.5  98.5  93.1  98.5  |  67.5  67.5  68.8  67.5  67.6  |
+windows-1253            36846  |   2.0  72.1  86.9  74.6  74.6  |   0.1  89.5  89.5  89.1  92.0  |  77.3  77.3  80.7  80.5  80.5  |
+windows-1254            36688  |   5.3  57.8  84.0  79.3  79.3  |   0.0   0.0   0.0  39.1  43.5  |  23.3  23.3  24.4  49.3  49.3  |
+windows-1255            31171  |   6.9  34.4  48.5  34.8  35.9  |  93.9  95.7  95.7  96.7  97.3  |  76.0  76.0  79.0  77.3  77.3  |
+windows-1256            41955  |  36.2  60.3  80.0  37.6  37.6  |   0.0   0.0   0.0   1.3   1.4  |  62.2  62.2  72.4  63.5  63.5  |
+windows-1257            30902  |   0.0   0.0   0.0  47.4  47.6  |   0.0   0.0   0.0  43.7  49.4  |  13.9  13.9  16.2  50.3  50.4  |
+windows-1258            36850  |   0.0   0.0   0.0   5.3   5.4  |   0.0   0.0   0.0   5.2   5.5  |  57.3  57.3  57.4  62.5  62.5  |
+windows-874             31260  |   0.0   0.0   0.0   9.1   9.1  |   0.0   0.0   0.0  83.7  94.7  |  60.4  60.4  69.3  69.1  69.1  |
+x-EUC-TW                26571  |   0.0   0.0   0.0   4.2   4.4  |  43.8  43.8  43.8  48.1  48.3  |  83.9  83.9  85.1  88.1  88.3  |
+x-MacRoman               1963  |   0.0   0.0   0.0  53.9  54.2  |   0.0   0.0   0.0  54.6  54.9  |   6.0   6.0   8.3  60.3  60.8  |
+x-mac-cyrillic           1742  |   0.0   0.0   0.0   0.0   0.0  |  52.0  52.0  52.0  52.0  52.0  |  36.0  36.0  39.1  36.0  36.0  |
+x-windows-949           36726  |   0.0   0.0  28.6   2.8   3.0  |   0.0  83.7  83.7  86.5  86.8  |  74.8  74.8  91.2  77.7  77.9  |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+OVERALL                 1220660  |  20.5  42.2  52.9  42.3  42.5  |  22.0  33.4  33.4  45.8  46.8  |  47.8  50.1  55.1  61.0  61.0  |
+  Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
+  µs/sample                    |                       26.1  |                        7.5  |                        2.6  |
+