encoding-detectors.adoc

Configuring Encoding Detectors

Tika uses a chain of encoding detectors to determine the character encoding of plain text and HTML content. DefaultEncodingDetector loads detectors via the Java service-provider interface (SPI) and runs them in registration order. See Charset Detection Pipeline for design details.

Default Detection Chain

With the stock dependencies on the classpath:

Step	Detector	Returns non-null when…
1	bom-detector	A UTF-8, UTF-16 LE/BE, or UTF-32 LE/BE byte-order mark is present. Emits DECLARATIVE.
2	metadata-charset-detector	A charset= parameter is present in the Content-Type metadata field (populated from an HTTP response header or similar). Emits DECLARATIVE.
3	mojibuster-encoding-detector	A structural UTF-32 check, structural UTF-16 specialist, UTF-8 grammar gate, and 33-class byte-bigram Naive Bayes classifier. STRUCTURAL for structural hits; STATISTICAL for NB predictions.
4	html-encoding-detector	An HTML <meta charset="…"> or <meta http-equiv="Content-Type"> tag is found (fast lenient regex matcher, curated WHATWG label aliases). Emits DECLARATIVE.
5	junk-filter-encoding-detector	MetaEncodingDetector — strips HTML/XML markup, decodes the probe under each candidate charset, and picks the cleanest decoding via a script-aware text-quality model. Honours a declaration when its decoding is byte-identical to at least one other candidate’s. Always runs last.

Note	junk-filter-encoding-detector is supplied by tika-ml-junkdetect and SPI-loads when the module jar is on the classpath. Omit it for plain first-match-wins — see Strict first-match-wins (no JunkFilter arbitration).

Available Detectors

All detectors implement org.apache.tika.detect.EncodingDetector and can be referenced by their SPI name in JSON configuration.

Name	Module	Description
bom-detector	tika-core	Reads the first 4 bytes for BOM signatures. In the default chain.
metadata-charset-detector	tika-core	Reads declarative hints (Content-Type charset, Content-Encoding) from the Metadata object. Applies WHATWG label normalization (ISO-8859-1 and US-ASCII → windows-1252). In the default chain.
mojibuster-encoding-detector	tika-encoding-detector-mojibuster	Byte-bigram Naive Bayes classifier plus structural detectors for UTF-32 and UTF-16 and a UTF-8 grammar gate. 33 classes including CJK multi-byte, EBCDIC variants, DOS code pages, Cyrillic, Windows single-byte, ISO-8859-3/16, Mac, and UTF-8. In the default chain. See the design doc.
html-encoding-detector	tika-encoding-detector-html	Fast lenient regex matcher for <meta charset> / http-equiv tags, with a curated subset of WHATWG label aliases. In the default chain.
standard-html-encoding-detector	tika-encoding-detector-html	Spec-strict WHATWG prescan algorithm. Not in the default chain — opt in explicitly if you need strict WHATWG tokenisation (e.g. ignoring charset declarations inside comments or other contexts the lenient regex may match).
junk-filter-encoding-detector	tika-ml-junkdetect	Text-quality arbitrator (MetaEncodingDetector). In the default chain when the module jar is on the classpath; always runs last.
icu4j-encoding-detector	tika-encoding-detector-icu4j	Wraps ICU4J CharsetDetector. Legacy — the NB pipeline supersedes it for most cases. Not auto-registered.
universal-encoding-detector	tika-encoding-detector-universal	State-machine structural prober (juniversalchardet fork). Not auto-registered; opt in if you specifically need it.

Configuration Examples

Exclude a detector from the default chain

{
  "encoding-detectors": [
    {
      "default-encoding-detector": {
        "exclude": ["bom-detector"]
      }
    }
  ]
}

Strict first-match-wins (no JunkFilter arbitration)

Omit JunkFilter to get plain first-match-wins (each base detector’s top result wins in registration order):

{
  "encoding-detectors": [
    {"bom-detector": {}},
    {"metadata-charset-detector": {}},
    {"html-encoding-detector": {}},
    {"mojibuster-encoding-detector": {}}
  ]
}

Trade-off: lying declarations propagate unfiltered, and Mojibuster’s statistical guess wins ties without any text-quality cross-check. See the design doc.

Configure the HTML detector’s read limit

html-encoding-detector reads up to 65 536 bytes by default when scanning for the <meta charset> tag. Raise it if your documents embed large <script> blocks before the meta tag (TIKA-2485):

{
  "encoding-detectors": [
    {"bom-detector": {}},
    {"metadata-charset-detector": {}},
    {
      "html-encoding-detector": {
        "markLimit": 131072
      }
    },
    {"mojibuster-encoding-detector": {}},
    {"junk-filter-encoding-detector": {}}
  ]
}

Legacy chain (ICU4J + juniversalchardet)

Not recommended — the NB pipeline is strictly better on accuracy and latency — but available for regression testing or comparison:

{
  "encoding-detectors": [
    {"bom-detector": {}},
    {"metadata-charset-detector": {}},
    {"html-encoding-detector": {}},
    {"icu4j-encoding-detector": {}},
    {"universal-encoding-detector": {}}
  ]
}