TIKA-4720: merge and clean

Author: tballison

docs/modules/ROOT/nav.adoc

@@ -47,7 +47,6 @@
 * xref:advanced/index.adoc[Advanced]
 ** xref:advanced/charset-detection-design.adoc[Charset Detection Pipeline]
 ** xref:advanced/language-detection.adoc[Language Detection]
-** xref:advanced/generative-language-model.adoc[Generative Language Model]
 ** xref:advanced/language-detection-build.adoc[Building the Language Detector]
 ** xref:advanced/junk-detection.adoc[Text Quality Scoring (Junk Detection)]
 ** xref:advanced/junk-detection-build.adoc[Building the Junk Detector]

docs/modules/ROOT/pages/advanced/charset-detection-design.adoc

@@ -54,19 +54,16 @@ is found (fast lenient regex matcher, curated WHATWG label aliases).
 Emits DECLARATIVE.
 
 |5
-|`charsoup-encoding-detector`
-|`MetaEncodingDetector` — arbitrates across all base-detector candidates
-using a character-bigram language model.  When one or more base detectors
-return a result, CharSoup decodes the probe with each candidate charset
-and picks the one whose decoded text scores best for any supported
-language.  Always runs last.
+|`junk-filter-encoding-detector`
+|`MetaEncodingDetector` — strips HTML/XML markup, decodes the probe under
+each candidate charset, and picks the cleanest decoding via a script-aware
+text-quality model.  Honours a declaration when its decoding is
+byte-identical to at least one other candidate's.  Always runs last.
 |===
 
-NOTE: `charsoup-encoding-detector` is supplied by
-`tika-encoding-detector-charsoup` and loads automatically via SPI when
-the module jar is present.  Omit it to get strict first-match-wins
-behaviour that honours declarations authoritatively — see
-<<opting-out-of-charsoup>>.
+NOTE: `junk-filter-encoding-detector` is supplied by `tika-ml-junkdetect`
+and SPI-loads when the module jar is on the classpath.  Omit it for plain
+first-match-wins — see <<opting-out-of-arbitration>>.
 
 == Available Detectors
 
@@ -106,15 +103,10 @@ curated subset of WHATWG label aliases.  In the default chain.
 explicitly if you need strict WHATWG tokenisation (e.g. ignoring charset
 declarations inside comments or other contexts the lenient regex may match).
 
-|`charsoup-encoding-detector`
-|`tika-encoding-detector-charsoup`
-|Language-aware arbitrator (`MetaEncodingDetector`).  In the default
-chain when the module jar is on the classpath; always runs last.
-
-|`mojibuster-encoding-detector`
-|`tika-encoding-detector-mojibuster`
-|Legacy maxent charset classifier.  Still available for explicit opt-in;
-not auto-registered.
+|`junk-filter-encoding-detector`
+|`tika-ml-junkdetect`
+|Text-quality arbitrator (`MetaEncodingDetector`).  In the default chain
+when the module jar is on the classpath; always runs last.
 
 |`icu4j-encoding-detector`
 |`tika-encoding-detector-icu4j`
@@ -144,13 +136,11 @@ auto-registered; opt in if you specifically need it.
 }
 ----
 
-[[opting-out-of-charsoup]]
-=== Strict declared-charset honoring (no CharSoup arbitration)
+[[opting-out-of-arbitration]]
+=== Strict first-match-wins (no JunkFilter arbitration)
 
-By default, CharSoup arbitrates across all candidates including DECLARATIVE
-ones — which means a `<meta charset>` tag is not automatically
-authoritative.  If you want declarations to win unconditionally (HTML5-spec
-behaviour), omit CharSoup:
+Omit JunkFilter to get plain first-match-wins (each base detector's top
+result wins in registration order):
 
 [source,json]
 ----
@@ -164,11 +154,10 @@ behaviour), omit CharSoup:
 }
 ----
 
-Without CharSoup, `CompositeEncodingDetector` runs in first-match-wins
-mode.  Trade-off: lying declarations (common on the legacy web)
-propagate unfiltered.  See
-xref:../advanced/charset-detection-design.adoc#opting-out-of-arbitration[the design doc]
-for details.
+Trade-off: lying declarations propagate unfiltered, and Mojibuster's
+statistical guess wins ties without any text-quality cross-check.
+See
+xref:../advanced/charset-detection-design.adoc#opting-out-of-arbitration[the design doc].
 
 === Configure the HTML detector's read limit
 
@@ -188,7 +177,7 @@ large `<script>` blocks before the meta tag (TIKA-2485):
       }
     },
     {"mojibuster-encoding-detector": {}},
-    {"charsoup-encoding-detector": {}}
+    {"junk-filter-encoding-detector": {}}
   ]
 }
 ----

docs/modules/ROOT/pages/advanced/generative-language-model.adoc

@@ -63,6 +63,11 @@
       <artifactId>tika-langdetect-charsoup</artifactId>
       <version>${project.version}</version>
     </dependency>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>tika-ml-junkdetect</artifactId>
+      <version>${project.version}</version>
+    </dependency>
     <dependency>
       <groupId>${project.groupId}</groupId>
       <artifactId>tika-xmp</artifactId>

docs/modules/ROOT/pages/configuration/encoding-detectors.adoc

@@ -138,6 +138,11 @@
         <artifactId>tika-langdetect-charsoup</artifactId>
         <version>${revision}</version>
       </dependency>
+      <dependency>
+        <groupId>org.apache.tika</groupId>
+        <artifactId>tika-ml-junkdetect</artifactId>
+        <version>${revision}</version>
+      </dependency>
       <dependency>
         <groupId>org.apache.tika</groupId>
         <artifactId>tika-langdetect-test-commons</artifactId>

tika-app/pom.xml

@@ -37,9 +37,10 @@
  * and takes priority over all statistical or structural inference.
  *
  * <p>SPI-loaded first in the default encoding-detector chain so that BOM evidence
- * reaches {@code CharSoupEncodingDetector} before any statistical detector runs.
- * {@code MojibusterEncodingDetector} strips the BOM from its own probe independently
- * to ensure consistent model inference (BOMs are excluded from training data).</p>
+ * reaches the meta arbiter ({@code JunkFilterEncodingDetector}) before any
+ * statistical detector runs.  {@code MojibusterEncodingDetector} strips the BOM
+ * from its own probe independently to ensure consistent model inference (BOMs are
+ * excluded from training data).</p>
  *
  * @since Apache Tika 0.x (moved to org.apache.tika.detect in 4.0)
  */

tika-bom/pom.xml

@@ -44,8 +44,8 @@
  *
  * <p>This detector is SPI-loaded in {@code tika-core} and therefore always present
  * in the default encoding-detector chain.  Its DECLARATIVE result is visible to
- * {@code CharSoupEncodingDetector}, which can weigh it against structural or
- * statistical evidence from other detectors.</p>
+ * {@code JunkFilterEncodingDetector}, which honours it when its decoding is
+ * byte-identical to at least one other candidate's.</p>
  *
  * @since Apache Tika 4.0
  */