Merge remote-tracking branch 'origin/main' into TIKA-4816-metadata-key-api

Author: tballison

.github/workflows/main-jdk17-build.yml

@@ -65,4 +65,7 @@ jobs:
           java-version: ${{ matrix.java }}
           cache: 'maven'
       - name: Run E2E Tests
-        run: mvn -pl tika-e2e-tests -am clean verify -Pe2e -B "-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
+        # -pl must name the leaf modules: tika-e2e-tests is an aggregator pom, and Maven
+        # does not pull in a selected aggregator's children, so this job built two pom-only
+        # modules and ran zero tests while reporting green.
+        run: mvn -pl :tika-e2e-tests-server,:tika-grpc-e2e-test -am clean verify -Pe2e -B "-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"

CHANGES.txt

@@ -30,6 +30,79 @@ Release 4.0.0 - ???
      API cleanups (cf: prefix, writelimiter package, smaller removals) are
      enumerated in the migration guide, migrating-to-4x.adoc (TIKA-4816).
 
+   * tika-server: the /translate endpoints have been removed (TIKA-4809).
+
+   * tika-server: /detect/stream and /language/stream and /language/string have
+     been collapsed into /detect and /language (TIKA-4809).
+
+   * tika-server: caller errors now map to accurate HTTP status codes instead of
+     always returning 200 or 500. A saturated worker pool returns 429, a
+     crashed/timed-out/OOM worker returns 503, an unknown or reserved
+     fetcher/emitter or bad handler returns 400, and an over-limit body returns
+     413; the 429 and 503 responses carry a Retry-After header (TIKA-4809).
+
+   * tika-server: the bare /tika endpoint now returns Markdown too (was XHTML),
+     completing the Markdown default (TIKA-4663); use /tika/xml for XHTML (TIKA-4809).
+
+   * tika-server: request bodies are now capped by maxRequestSizeBytes,
+     defaulting to 1 GiB; larger requests are rejected with 413, including
+     over-limit chunked uploads, which previously surfaced as an empty 500
+     (TIKA-4809).
+
+   * tika-server: a /meta request with no Accept header now returns JSON;
+     3.x returned CSV. CSV is still available via Accept: text/csv (TIKA-4809).
+
+   * tika-server: the raw /tika family's 422 responses carry the extracted
+     content only; the exception is no longer appended to the body -- use
+     /rmeta for the structured exception (TIKA-4809).
+
+   * tika-server: /async validates fetcher/emitter ids at POST time (400),
+     rejects a batch larger than the queue's total capacity with 400 instead
+     of throttling it, and one bad tuple no longer stops the async workers
+     (TIKA-4809).
+
+   * tika-server: /pipes returns 400 with the reason for a malformed request
+     body and rejects emit strategies other than EMIT_ALL, whose passed-back
+     data the /pipes response cannot carry (TIKA-4809).
+
+   * tika-server: the 'endpoints' allowlist now also gates SPI-provided
+     resources; a discovered resource binds only when its root endpoint is
+     enabled (TIKA-4809).
+
+   * FetchEmitTuple JSON now names the per-tuple parse context "parse-context"
+     (was "parseContext") and rejects unknown tuple fields with an error naming
+     the field (TIKA-4809).
+
+   * The pipes config keys staleFetcherTimeoutSeconds and
+     staleFetcherDelaySeconds have been removed; a config still carrying them
+     fails startup (TIKA-4809).
+
+   * ExceptionUtils.trimMessage has been removed from tika-core; it moved into
+     tika-eval-core (TIKA-4809).
+
+   * BasicContentHandlerFactory.parseHandlerType now throws
+     IllegalArgumentException for an unrecognized handler name instead of
+     silently returning the supplied default (TIKA-4809).
+
+   * HttpClientFactory's inert redirect-host allowlist accessors
+     (get/setAllowedHostsForRedirect) have been removed (TIKA-4809).
+
+   * TimeoutLimits: progressTimeoutMillis of 0 combined with a positive
+     totalTaskTimeoutMillis is now rejected at config load; it would kill
+     every task immediately (TIKA-4809).
+
+   * The http-fetcher now verifies TLS certificates and hostnames by default;
+     set verifySsl:false to opt out (TIKA-4809).
+
+   * tika-server: /pipes returns the same JSON body as /tika/rmeta/unpack --
+     {"status":<RESULT_STATUS>,"message":...} -- instead of a /pipes-only
+     {"status":"ok"|"process_crash",...} shape (TIKA-4809).
+
+   * tika-server: POST /async requires an object body {"tuples":[...]} instead of
+     a bare JSON array (TIKA-4809).
+
+   * tika-server: the tika-server-client module has been removed (TIKA-4809).
+
   NEW FEATURES
 
    * Content-based detection of ASN.1/DER crypto containers at parse time. An
@@ -41,6 +114,13 @@ Release 4.0.0 - ???
    * MagicDetector now compiles its regular expression once, in the
      constructor, instead of recompiling it on every match (TIKA-4796).
 
+   * Unified timeout model across the library, pipes, and server: a total-task
+     budget plus a progress/stall timeout, composed recursively over embedded
+     documents (TIKA-4813).
+
+   * tika-server: /meta now runs through the same pipes-backed parser as the
+     other extraction endpoints (TIKA-4809).
+
    * tika-grpc's fetchAndParseServerSideStreaming now completes the call
      after delivering its reply, instead of leaving the client waiting
      for a terminal signal that never came (TIKA-4804).

docs/modules/ROOT/pages/advanced/integration-testing/run-uat-script.adoc

@@ -36,20 +36,20 @@ release-tools/uat/
 
 == What it covers
 
-Roughly 25 REST endpoint checks across the default-mode endpoints, header
+Roughly 30 REST endpoint checks across the default-mode endpoints, header
 behavior, and error handling — the same surface enumerated in the manual
 walkthrough at xref:advanced/integration-testing/tika-server.adoc[Tika-Server
 Integration Testing], translated to bash + curl assertions.
 
 Coverage includes:
 
 * `/version`, `/parsers`, `/detectors`, `/mime-types` (introspection)
-* `/detect/stream` (mime detection)
+* `/detect` (mime detection)
 * `/tika`, `/tika/text`, `/tika/xml`, `/tika/json` (parse)
 * `/meta`, `/meta/\{field}` (metadata)
 * `/rmeta`, `/rmeta/text` (recursive metadata)
 * `/unpack/all` (embedded extraction; verifies the response is a valid zip)
-* `/language/stream`
+* `/language`
 * `/meta/form`, `/rmeta/form` (multipart variants)
 * `allowPerRequestConfig=false` gating, both enforcement points: the path-based
   filter (`/meta/config`, `/rmeta/config`, `/tika/config`, `/unpack/all/config` all
@@ -59,6 +59,14 @@ Coverage includes:
 * `/status` is not registered by default (returns 404 unless explicitly listed
   under `endpoints`)
 * `X-Tika-OCRskipOcr` header
+* Handler selection: `/tika/text` is body-only (no `<title>` in the output),
+  `/tika/json` with no handler named matches `/tika/json/md` (markdown is the
+  default), and an unrecognized handler name returns 400 rather than silently
+  falling back
+* `/meta` with `Accept: application/rdf+xml` returns XMP -- content negotiation,
+  not a distinct path. Worth a smoke check because this shipped broken in
+  4.0.0-alpha-1 and beta-1: the resource failed to load and the endpoint 406'd
+* Removed endpoints stay removed (`/translate/*` returns 404)
 * 404 / 405 error handling
 * OCR (`PUT /tika/text` of an image whose text only exists as pixels): skipped when the
   server has no tesseract (the minimal image / a plain `java -jar` server), and a hard

docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc

@@ -244,7 +244,7 @@ Create `/tmp/tika-app-test/my-config.json`:
     "basic-content-handler-factory": {
       "type": "TEXT",
       "writeLimit": 100000,
-      "throwOnWriteLimitReached": false
+      "throwOnWriteLimit": false
     }
   },
   "parsers": [

docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc

@@ -57,11 +57,11 @@ curl -s http://localhost:9998/version
 
 *Expected:* `Apache Tika X.X.X`
 
-=== Test 2: PUT /detect/stream
+=== Test 2: PUT /detect
 
 [source,bash]
 ----
-curl -s -X PUT -T testPDF.pdf http://localhost:9998/detect/stream
+curl -s -X PUT -T testPDF.pdf http://localhost:9998/detect
 ----
 
 *Expected:* `application/pdf`
@@ -118,7 +118,8 @@ curl -s -X PUT -H "Accept: application/json" -T testPDF.pdf http://localhost:999
 curl -s -X PUT -T testPDF.pdf http://localhost:9998/meta/Content-Type
 ----
 
-*Expected:* `Content-Type,application/pdf`
+*Expected:* JSON containing only the requested field: `{"Content-Type":"application/pdf"}`
+(the header-less default is JSON; add `-H "Accept: text/csv"` for `Content-Type,application/pdf`)
 
 === Test 9: PUT /rmeta
 
@@ -138,11 +139,11 @@ curl -s -X PUT -T test_recursive_embedded.docx http://localhost:9998/rmeta/text
 
 *Expected:* JSON array with ToTextContentHandler content.
 
-=== Test 11: PUT /language/stream
+=== Test 11: PUT /language
 
 [source,bash]
 ----
-curl -s -X PUT -T testPDF.pdf http://localhost:9998/language/stream
+curl -s -X PUT -T testPDF.pdf http://localhost:9998/language
 ----
 
 *Expected:* Two-letter language code (e.g., `en`, `th`).
@@ -155,7 +156,8 @@ curl -s -X PUT -T test_recursive_embedded.docx http://localhost:9998/unpack/all
 unzip -l /tmp/unpack.zip
 ----
 
-*Expected:* ZIP file containing extracted embedded files plus `__TEXT__` and `__METADATA__` files.
+*Expected:* ZIP file containing the extracted embedded files, a `*.metadata.json` entry
+per file, and the original container document.
 
 === Test 13: GET /parsers
 
@@ -406,7 +408,7 @@ The following endpoints were tested and verified working:
 |Endpoint |Method |Status
 
 |`/version` |GET |PASS
-|`/detect/stream` |PUT |PASS
+|`/detect` |PUT |PASS
 |`/tika` |PUT |PASS
 |`/tika/text` |PUT |PASS
 |`/tika/html` |PUT |PASS
@@ -416,7 +418,7 @@ The following endpoints were tested and verified working:
 |`/meta/\{field}` |PUT |PASS
 |`/rmeta` |PUT |PASS
 |`/rmeta/text` |PUT |PASS
-|`/language/stream` |PUT |PASS
+|`/language` |PUT |PASS
 |`/unpack/all` |PUT |PASS
 |`/parsers` |GET |PASS
 |`/detectors` |GET |PASS
@@ -445,7 +447,7 @@ The following endpoints were tested and verified working:
 
 === Issue 1: Language Detection Accuracy
 
-Short texts may not be detected reliably. The `/language/stream` endpoint works best with substantial text content.
+Short texts may not be detected reliably. The `/language` endpoint works best with substantial text content.
 
 == Quick Reference
 
@@ -478,7 +480,7 @@ curl -X PUT -T file.docx http://localhost:9998/rmeta/text
 === Detection
 [source,bash]
 ----
-curl -X PUT -T file.pdf http://localhost:9998/detect/stream
+curl -X PUT -T file.pdf http://localhost:9998/detect
 ----
 
 === Extract Embedded Files