Merge pull request #2997 from apache/TIKA-4809-stage-3

Author: tballison

docs/modules/ROOT/pages/advanced/setting-limits.adoc

@@ -52,7 +52,8 @@ This is the same configuration tested in `AllLimitsTest.java`:
       "zipBombRatio": 100
     },
     "timeout-limits": {
-      "taskTimeoutMillis": 60000
+      "totalTaskTimeoutMillis": 3600000,
+      "progressTimeoutMillis": 60000
     },
     "standard-metadata-limiter-factory": {
       "maxTotalBytes": 1048576,
@@ -243,27 +244,71 @@ See test: `tika-serialization/src/test/java/org/apache/tika/config/OutputLimitsT
 
 == Timeout Limits
 
-The `TimeoutLimits` class controls time-based limits for parsing operations.
+The `TimeoutLimits` class controls time-based limits for parsing operations. Tika 4.x
+uses a *two-tier* timeout: one bound on total wall-clock time, and one on time elapsed
+since the parser last reported progress.
 
 === Configuration Options
 
 [cols="2,1,3"]
 |===
 |Setting |Default |Description
 
-|`taskTimeoutMillis`
-|60000 (1 minute)
-|Maximum time in milliseconds for a parse operation to complete.
+|`totalTaskTimeoutMillis`
+|3600000 (1 hour)
+|Maximum wall-clock time in milliseconds for the entire parse task.
+
+|`progressTimeoutMillis`
+|120000 (2 minutes)
+|Maximum time in milliseconds since the parser last reported progress. Catches
+infinite loops and hung processes.
 |===
 
+[IMPORTANT]
+====
+Which bound actually applies depends on whether the parser reports progress.
+A parser that never calls `TikaProgressTracker.update()` never advances the
+timer, so it effectively gets `progressTimeoutMillis` as its total timeout —
+matching the single-timeout behavior of earlier versions.
+
+In practice only long-running parsers report progress: `TesseractOCRParser`,
+`Tess4JParser`, `ExternalParser`, `GDALParser`, the VLM and image-embedding
+parsers, and `StringsParser`. Everything else — including container parsing and
+embedded-document recursion — does not.
+
+Those parsers report progress *after* each external-process invocation
+completes, not while one is running. So a document that needs many OCR calls
+can extend well past `progressTimeoutMillis`, because each finished page resets
+the timer — but a *single* call that runs longer than `progressTimeoutMillis`
+is still cut short.
+
+Because of this, `progressTimeoutMillis` also caps how long any single external
+process may run. Parsers that spawn processes size their own timeout via
+`TimeoutLimits.getProcessTimeoutMillis(context, ...)`, which never allows a
+value beyond `progressTimeoutMillis`, so the process is stopped just before the
+progress watchdog would fire.
+
+The shipped defaults are aligned: `progressTimeoutMillis` is 120 seconds and
+the bundled process-spawning parsers (OCR, strings, inference) each default to
+a 120-second per-process timeout, so those defaults are reachable. **If you
+raise a per-process timeout above 120 seconds, raise `progressTimeoutMillis`
+to match** — raising the parser's own timeout alone has no effect.
+
+For most documents — anything without one of the parsers above in the chain —
+the effective ceiling is `progressTimeoutMillis`, not `totalTaskTimeoutMillis`.
+Lower `totalTaskTimeoutMillis` if you need a hard ceiling on OCR-heavy or
+external-process work regardless of progress.
+====
+
 === JSON Configuration
 
 [source,json]
 ----
 {
   "parse-context": {
     "timeout-limits": {
-      "taskTimeoutMillis": 120000
+      "totalTaskTimeoutMillis": 7200000,
+      "progressTimeoutMillis": 120000
     }
   }
 }
@@ -275,7 +320,7 @@ Configuration file: `tika-serialization/src/test/resources/configs/timeout-limit
 
 [source,java]
 ----
-TimeoutLimits limits = new TimeoutLimits(120000);
+TimeoutLimits limits = new TimeoutLimits(7200000, 120000);
 context.set(TimeoutLimits.class, limits);
 
 // Helper method

docs/modules/ROOT/pages/developers/serialization.adoc

@@ -241,6 +241,26 @@ The serialization system implements a security allowlist:
 This prevents attacks where malicious JSON specifies dangerous classes
 for instantiation.
 
+[IMPORTANT]
+====
+The allowlist governs *which components may be instantiated* from JSON. It does
+not restrict *how an already-loaded component may be configured*.
+
+Self-configuring components — which includes every `Parser`, since `Parser`
+extends `SelfConfiguring` — are skipped by the wire-block scan
+(`ParseContextDeserializer.assertNoBlockedComponents`): their config subtree is
+passed through to the component unexamined. So while a request cannot bind a new
+`Parser` from the wire, a request carrying
+`{"parse-context": {"pdf-parser": {"ocr": {"strategy": "OCR_AND_TEXT_EXTRACTION"}}}}`
+will reach `PDFParser` and take effect.
+
+That is why per-request configuration is gated separately by
+`allowPerRequestConfig`, which is off by default. Treat "the caller may supply
+per-request config" as equivalent to "the caller may set any parser option,
+including options that spawn external processes such as OCR" — not as something
+the allowlist constrains.
+====
+
 [source,java]
 ----
 // This will FAIL - class not registered

docs/modules/ROOT/pages/migration-to-4x/design-notes-4x.adoc

@@ -74,6 +74,14 @@ may be bound from the wire; `Parser`, `Detector`, `Renderer`, and similar are
 blocked before anything is constructed. See
 xref:developers/serialization.adoc[Serialization and Configuration].
 
+Note the boundary: the allowlist blocks *binding a component* from the wire, not
+*configuring one that is already loaded*. Self-configuring components — every
+`Parser` among them — have their config subtree passed through unscanned, so a
+per-request config can still set parser options (including ones that spawn
+external processes, such as OCR). This is why `allowPerRequestConfig` is a
+separate gate and is off by default; the allowlist alone does not make
+per-request configuration safe to expose.
+
 === Implementation Challenges
 
 * Converted code to true Java beans with matching getters/setters

docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc

@@ -135,6 +135,31 @@ for error text should check `tk:exception:container-exception` (full-object
 endpoints) or the `422` body (`/meta/\{field}`, populated only when
 `returnStackTrace=true`).
 
+Two changes to the returned metadata come with this, neither of which produces an
+error:
+
+* **`/meta` no longer returns a `language` field.** Language detection previously ran
+  inline on this endpoint via a dedicated content handler that buffered text solely to
+  detect the language, which meant holding the document text twice to populate one
+  field. That handler was removed. `/meta` deliberately parses with the `ignore`
+  content handler, so there is no text for a language detector to work from.
++
+**Migration:** configure a language-detection metadata filter
+(`charsoup-metadata-filter`, `optimaize`, or `opennlp`) and use `/rmeta` or
+`/tika/json`, which capture content. The detected value arrives as
+`tk:detected-language`, with `tk:detected-language-confidence`. Note that these
+filters read `tk:content`, so they are no-ops on `/meta` and on any endpoint
+configured with the `ignore` handler.
+
+* **`/meta` now sets `tk:exception:embedded-depth-limit-reached` on any document
+  with embedded content.** `/meta` suppresses embedded parsing by setting an embedded
+  depth limit of `0`, and reaching a limit is recorded. The previous implementation
+  suppressed embedded documents by a different mechanism that recorded nothing. The
+  flag is expected on this endpoint and does not indicate a truncated result.
++
+**Migration:** clients that alert on the presence of any `tk:exception:*` key should
+exclude this one for `/meta`.
+
 === Accept Header Routing Removed
 
 The `/tika` endpoint no longer routes based on `Accept` headers. Use explicit paths instead:
@@ -157,7 +182,24 @@ The following `TikaServerConfig` options have been removed:
 
 === `/pipes` and `/async` Require `allowPipes`; Per-Request Config Requires `allowPerRequestConfig`
 
-Previously these endpoints (and per-request parser configuration) were enabled simply by listing endpoints under `server.endpoints`. The capabilities are now split into two default-`false` flags in the `server` section:
+This replaces the `enableUnsecureFeatures` flag that alpha-1 briefly used, and before
+that, enabling these capabilities simply by listing endpoints under `server.endpoints`.
+`enableUnsecureFeatures` no longer exists: a config that still carries it fails to start
+with an "Unrecognized field" error naming the key, rather than silently ignoring it.
+The single flag has been split into two, so that granting batch/fetcher access and
+granting per-request parser configuration are separate decisions:
+
+|===
+|Was |Now
+
+|`enableUnsecureFeatures: true` (to use `/pipes` or `/async`)
+|`allowPipes: true`
+
+|`enableUnsecureFeatures: true` (to send per-request config)
+|`allowPerRequestConfig: true`
+|===
+
+The capabilities are two default-`false` flags in the `server` section:
 
 * `allowPipes` gates the `/pipes` and `/async` endpoints, which drive process-isolated batch parsing through your fetchers and emitters. Selecting either without `allowPipes` causes the server to refuse to start with a clear error.
 * `allowPerRequestConfig` gates per-request parser configuration: the `/config` family of endpoints and the multipart `config` part. When off, such requests are rejected with 403.
@@ -189,7 +231,7 @@ All tika-server configurations must now include a `pipes` section and a `file-sy
   "fetchers": {
     "file-system-fetcher": {
       "file-system-fetcher": {
-        "allowAbsolutePaths": true
+        "basePath": "/path/to/your/input"
       }
     }
   },
@@ -205,6 +247,23 @@ All tika-server configurations must now include a `pipes` section and a `file-sy
 }
 ----
 
+[IMPORTANT]
+====
+Set `basePath` to a directory that contains only the documents you intend the
+server to read. It is the filesystem sandbox: the fetcher rejects any fetch key
+that resolves outside it, including absolute paths and `../` traversal, and
+re-checks after resolving symlinks.
+
+Setting `allowAbsolutePaths` instead of `basePath` turns that sandbox off
+entirely — fetch keys are then used as raw absolute paths, so any caller who can
+reach `/pipes` can read any file the server process can read. The matching
+emitter setting is worse: it grants arbitrary file *write*. `allowAbsolutePaths`
+is not a relaxation of `basePath`; it is what you get when there is no
+`basePath` at all, and it is a no-op when `basePath` is set. Use it only if you
+genuinely intend an unsandboxed fetcher and have restricted access to the server
+by other means.
+====
+
 [IMPORTANT]
 ====
 `numClients` is not boilerplate to copy unchanged from this example. In 3.x,

docs/modules/ROOT/pages/pipes/plugins/filesystem.adoc

@@ -81,16 +81,16 @@ The outer key (`fsf`) is the fetcher ID — referenced by `pipesIterator.fetcher
 |Field |Default |Description
 
 |`basePath`
-|_required_
-|Base directory for fetch operations. Fetch keys are resolved relative to this path.
+|_none_
+|Base directory for fetch operations. Fetch keys are resolved relative to this path and must stay inside it. Not technically required, but omitting it disables containment entirely — see `allowAbsolutePaths` below and <<security-notes>>.
 
 |`extractFileSystemMetadata`
 |`false`
 |When `true`, attach file size, created, and modified timestamps to the metadata of each fetched document.
 
 |`allowAbsolutePaths`
 |`false`
-|When `true`, fetch keys may be absolute paths and `basePath` may be omitted. Use sparingly — see <<security-notes>>.
+|Permission to run *without* a `basePath`. It is not a relaxation of `basePath` — see <<security-notes>>.
 |===
 
 [#file-system-emitter]
@@ -253,6 +253,7 @@ Tradeoffs:
 [#security-notes]
 == Security Notes
 
-* **`basePath` is a sandbox boundary.** The fetcher and emitter reject fetch/emit keys that resolve outside `basePath`. Do not set `allowAbsolutePaths=true` unless the source of fetch keys is fully trusted — an attacker-controlled fetch key could otherwise read arbitrary files.
-* **Symlinks are followed.** A symlink under `basePath` pointing outside `basePath` may still be readable. If you need strict containment, do not allow symlinks in your input tree.
+* **`basePath` is the sandbox boundary, and it is the only one.** With `basePath` set, the fetcher and emitter reject any key that resolves outside it, including absolute paths and `../` traversal. `allowAbsolutePaths` has no effect in this state.
+* **Without `basePath` there is no containment at all.** The key is used as a raw absolute path, and the containment checks are skipped entirely. `allowAbsolutePaths=true` is how you assert that you intend this; it is a switch between two states, not a dial that loosens `basePath`. For the fetcher this means any file the process can read; for the emitter, any file it can write. Use it only when fetch/emit keys come from a fully trusted source and access to the service is restricted by other means.
+* **Symlink containment differs between fetcher and emitter.** The fetcher re-checks with `toRealPath()`, so a symlink under `basePath` pointing outside it is rejected. The emitter does not: it checks only the normalized path, so a symlink already present under its `basePath` can be written through. Do not rely on symlinks being contained on the emit side.
 * **Output directories are created automatically.** The emitter creates intermediate directories as needed. Make sure the process's umask is appropriate for the data being written.

docs/modules/ROOT/pages/pipes/plugins/http.adoc

@@ -92,7 +92,7 @@ include::example$pipes-http-fetcher.json[]
 
 |`maxRedirects`
 |`0`
-|Maximum number of redirects to follow. `0` means follow none.
+|Maximum number of redirects to follow. `0` means follow none. Not applied to range requests — see <<security-notes>>.
 
 |`maxSpoolSize`
 |`-1`
@@ -123,6 +123,38 @@ include::example$pipes-http-fetcher.json[]
 |Base64-encoded private key for asymmetric (RSA/ECDSA) JWT signing. Mutually exclusive with `jwtSecret`.
 |===
 
+[#security-notes]
+== Security Notes
+
+This fetcher makes the server issue HTTP requests to a URL supplied as the fetch key.
+That is a server-side request forgery primitive by design, and it is not constrained by
+this plugin. Treat the source of fetch keys as fully trusted, and restrict access to any
+endpoint that can reach it (`/pipes`, `/async`).
+
+* **The fetch key is used as the URL with no validation.** It is passed straight to
+  `new HttpGet(fetchKey)`. There is no scheme allowlist, no host denylist, and no check
+  against loopback, link-local, or RFC1918 addresses. A fetch key of
+  `http://169.254.169.254/...` reaches a cloud metadata endpoint like any other URL. The
+  resolved address is recorded in metadata *after* the fetch, not consulted before it.
+  Schemes other than `http`/`https` fail only because no other scheme is registered in
+  the connection manager — that is a side effect of the transport setup, not a check.
+
+* **TLS certificates are not verified, and this is not configurable here.** The
+  underlying client defaults to `verifySsl=false`, which installs an accept-everything
+  trust strategy and `NoopHostnameVerifier`. `HttpFetcherConfig` exposes no `verifySsl`
+  setting, so an http-fetcher config cannot turn verification on. Do not use this fetcher
+  to retrieve anything whose authenticity matters over an untrusted network.
+
+* **`maxRedirects` does not apply to range requests.** The main `fetch` builds a
+  `RequestConfig` from `maxRedirects`; the `startRange`/`endRange` overload sets no
+  request config at all and therefore uses the client's defaults (redirects enabled).
+  A `maxRedirects: 0` setting does not stop redirects on a range fetch.
+
+* **The redirect host allowlist is currently inert.** `CustomRedirectStrategy` will
+  refuse a redirect to a host outside `allowedHostsForRedirect`, but that set is never
+  populated from any configuration path, and the check is skipped when the set is empty.
+  Do not rely on it to contain redirects.
+
 [#notes]
 == Notes
 

docs/modules/ROOT/pages/pipes/timeouts.adoc

@@ -23,7 +23,9 @@ Tika Pipes uses a two-tier timeout system to handle both long-running tasks and
 
 * **`progressTimeoutMillis`** -- Maximum time between progress updates.
   If no progress is reported within this interval, the task is considered stalled and killed.
-  Default: `60000` (1 minute).
+  Default: `120000` (2 minutes).
+  This also caps how long any single external process (OCR, `ExternalParser`, VLM)
+  may run, since those parsers report progress only once a process completes.
 
 * **`totalTaskTimeoutMillis`** -- Maximum wall-clock time for an entire task.
   Even if the parser is making progress, the task is killed after this time.

docs/modules/ROOT/pages/security.adoc

@@ -56,8 +56,9 @@ and `allowComponentManagement` — the latter lets clients add, modify, and dele
 and read back stored configs, which can contain secrets — are off by default. Run it only behind
 network controls and, ideally, mutual TLS. See xref:using-tika/grpc/index.adoc[Tika gRPC].
 
-For the upgrade from the former `enableUnsecureFeatures` flag, see
-xref:migration-to-4x/migrating-tika-server-4x.adoc[Migrating tika-server to 4.x].
+For the upgrade from the former `enableUnsecureFeatures` flag, which is now split into
+`allowPipes` and `allowPerRequestConfig`, see
+xref:migration-to-4x/migrating-tika-server-4x.adoc#_pipes_and_async_require_allowpipes_per_request_config_requires_allowperrequestconfig[Migrating tika-server to 4.x].
 
 == Known Vulnerabilities