TIKA-4809: Document numClients's dual role, endpoint groups, and backpressure

Author: tballison

docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc

@@ -182,6 +182,24 @@ All tika-server configurations must now include a `pipes` section and a `file-sy
 }
 ----
 
+[IMPORTANT]
+====
+`numClients` is not boilerplate to copy unchanged from this example. In 3.x,
+`/tika`, `/rmeta`, and `/unpack` parsed in-process, in the request-handling
+JVM — no forked processes, no fixed concurrency limit. In 4.x, these same
+default-on endpoints *always* fork `numClients` child JVMs and share a fixed
+pool of that many concurrent workers. Size it too low for your request volume
+and callers start getting `429 CLIENT_UNAVAILABLE_WITHIN_MS` under load that
+used to just queue up on request threads instead (this is deliberate
+backpressure, not a bug — see
+xref:using-tika/server/index.adoc#_endpoints_and_forked_process_groups[Endpoints
+and Forked-Process Groups]). Size it too high for your host's core count, and
+the forked workers individually starve each other of CPU (see
+xref:pipes/cpu-sizing.adoc[Forked-JVM CPU Sizing]). Neither failure mode
+throws an error pointing at `numClients` as the cause — size it deliberately
+before deploying, not from this example.
+====
+
 == New Features
 
 === Process Isolation

docs/modules/ROOT/pages/pipes/cpu-sizing.adoc

@@ -112,6 +112,37 @@ Two `WARN`-level messages call out clearly-bad provisioning:
 
 `grep pipes-cpu-sizing` on the parent's logs surfaces all sizing-related output.
 
+[#_known_limitation_multiple_pipes_groups_in_one_process]
+== Known limitation: multiple Pipes groups in one process
+
+Everything above describes sizing for *one* `PipesParser` — one `pipes`
+config section, one set of forked workers. The auto-sizer has no visibility
+into anything else running in the same JVM.
+
+This matters concretely for tika-server: `/tika`+`/rmeta`+`/unpack` and
+`/pipes`+`/async` are backed by two *independent* `PipesParser` groups when
+both are enabled in the same server. Each group's auto-sizer computes its
+slice from `Runtime.availableProcessors()` as if it were the only consumer on
+the host — it does not know a sibling group in the same process is about to
+fork its own `numClients` workers too. The result: with `numClients=2` on
+both, you get 4 total forked JVMs, each capped assuming exclusive access to
+the whole host. Whether that's *actually* oversubscribed depends on your
+host's real core count relative to those combined `numClients` values — it's
+not automatic, but the auto-sizer also won't warn you, because each group
+looks correctly sized from its own perspective alone. See
+xref:using-tika/server/index.adoc#_endpoints_and_forked_process_groups[Endpoints
+and Forked-Process Groups] for the tika-server-specific guidance.
+
+The same applies to any application embedding `PipesForkParser`/`PipesParser`
+directly and constructing more than one instance in a single JVM — the
+auto-sizer will size each independently, with the same caveat.
+
+There is no automatic fix for this today: unlike the single-group case, where
+Tika detects and warns about bad provisioning, a *second* group has no way to
+learn what a sibling group already claimed. Mitigate it explicitly — either
+run only one group per process, or set `-XX:ActiveProcessorCount` yourself
+(next section) with the combined total in mind.
+
 == Disabling or overriding
 
 If you want to manage `ActiveProcessorCount` yourself (e.g., to allocate a

docs/modules/ROOT/pages/using-tika/server/index.adoc

@@ -33,6 +33,29 @@ request-handling JVM; treat those as best-effort under load. See
 xref:migration-to-4x/migrating-tika-server-4x.adoc[Migrating Tika Server to 4.x]
 for the full breaking-change list when upgrading from 3.x.
 
+[IMPORTANT]
+====
+This is not opt-in the way `/pipes` and `/async` are (those require
+`allowPipes=true` and refuse to start without it). `/tika`, `/rmeta`, and
+`/unpack` are **on by default** — the moment you run a basic `tika-server` and
+PUT a document to `/tika`, you are running Tika Pipes, with a real forked
+child process behind it. (`/meta` is the exception among the main
+content-extraction endpoints — it still parses in-process; see below.) If
+you're upgrading from 3.x, where these endpoints parsed in-process in a
+single JVM, this is a profound change: `pipes.numClients` now controls both
+how many requests these endpoints can serve concurrently and how many forked
+JVMs run at once, and it's easy to size it thinking about only one of those
+two things. Undersized for your request volume, and callers start waiting —
+then failing with `429`s — under load that used to just queue up on request
+threads in 3.x. Oversized for your host's core count, and the forked workers
+individually starve each other of CPU. Neither shows up as an error in your
+own code; both show up as "the server got slower" with nothing pointing at
+`numClients` as the cause. See
+<<_endpoints_and_forked_process_groups,Endpoints and Forked-Process Groups>>
+below and xref:pipes/cpu-sizing.adoc[Forked-JVM CPU Sizing] before deploying —
+don't treat the `numClients` value in example configs as safe boilerplate.
+====
+
 == Security
 
 IMPORTANT: The primary rule is *trusted callers only*. `tika-server` is not a security boundary:
@@ -196,10 +219,17 @@ a `message` field is also included (it often contains a server-side stack trace)
 |HTTP status |`status` values |Meaning
 
 |`503 Service Unavailable`
-|`TIMEOUT`, `OOM`, `UNSPECIFIED_CRASH`, `CLIENT_UNAVAILABLE_WITHIN_MS`
-|The forked parse process failed, or no parse client became available within the
-configured wait time (`CLIENT_UNAVAILABLE_WITHIN_MS`). The server is still healthy;
-the client may retry.
+|`TIMEOUT`, `OOM`, `UNSPECIFIED_CRASH`
+|The forked parse process actually failed (crashed, OOM'd, or exceeded its timeout).
+The server is still healthy; the client may retry.
+
+|`429 Too Many Requests`
+|`CLIENT_UNAVAILABLE_WITHIN_MS`
+|Nothing failed — no parse client became available within the configured wait time
+(deliberate backpressure, not a bug; see <<_endpoints_and_forked_process_groups,Endpoints
+and Forked-Process Groups>>). Distinct from `503` above on purpose: a `429` spike means
+"raise `numClients` or add capacity," a `503` spike means "something is actually
+crashing" — you can tell them apart from the status code alone.
 
 |`500 Internal Server Error`
 |`FAILED_TO_INITIALIZE`, `FETCH_EXCEPTION`, `EMIT_EXCEPTION`,
@@ -228,6 +258,10 @@ Server behavior beyond host/port is controlled by a JSON config file passed via
 |`false`
 |Opt-in for the `/pipes` and `/async` endpoints, which drive process-isolated fetching and parsing. The server refuses to start if either is selected without this flag (see <<_security_configuration,Security Configuration>>).
 
+|`endpoints`
+|_all defaults_
+|Which endpoints to expose. Leave unset to get the full default set (includes `/tika` and `/rmeta`). Explicitly listing endpoints also controls how many independent forked-process groups you run — see <<_endpoints_and_forked_process_groups,Endpoints and Forked-Process Groups>> below before combining `/tika`/`/rmeta` with `/pipes`/`/async`.
+
 |`allowPerRequestConfig`
 |`false`
 |Opt-in for per-request parser configuration: the `/config` family of endpoints and the multipart `config` part. When off, such requests are rejected with 403 (see <<_security_configuration,Security Configuration>>).
@@ -261,6 +295,64 @@ For the full Pipes-related sections (`pipes`, `fetchers`, `emitters`, `parse-con
 that tika-server 4.x requires, see
 xref:migration-to-4x/migrating-tika-server-4x.adoc#_configuration_changes[Configuration Changes].
 
+[#_endpoints_and_forked_process_groups]
+== Endpoints and Forked-Process Groups
+
+Two independent pipes-backed process groups exist, plus one endpoint that
+isn't pipes-backed at all:
+
+* **`/tika` + `/rmeta` + `/unpack`** share one group — all three go through
+the same `PipesParsingHelper`/`PipesParser`, sized by `pipes.numClients`.
+* **`/pipes` + `/async`** share a separate group (gated behind `allowPipes`),
+sized by the same `pipes.numClients` setting in the same config, but as an
+independent set of forked processes.
+* **`/meta` is not pipes-backed** — it still parses in-process, in the
+request-handling JVM, as in 3.x. It isn't bound by `numClients` and doesn't
+participate in anything below, but it also has no crash/OOM isolation: a
+hostile or pathological document sent to `/meta` can affect the
+request-handling process itself, unlike the pipes-backed endpoints where the
+same document only takes down a forked child. Treat `/meta` as best-effort
+under adversarial input.
+
+Within a pipes-backed group, `numClients` does two *separate* jobs, and it's
+worth understanding both before picking a value.
+
+=== It bounds how many requests that group can serve at once
+
+Each group holds a fixed pool of `numClients` workers. A request that arrives
+when all of them are busy doesn't fail immediately — it waits, up to
+`pipes.maxWaitForClientMillis` (default 60s), for one to free up. This is
+deliberate backpressure, not a bug: if a worker frees up in time, the request
+is served normally; if the wait times out, the server returns `429` with
+`status: CLIENT_UNAVAILABLE_WITHIN_MS` — an explicit "I'm at capacity, retry"
+signal, not a crash (see <<_error_responses,Error Responses>> above). Under
+3.x's in-process model there was no equivalent hard cap — requests just piled
+up on the HTTP server's own thread pool instead. If you're seeing
+`CLIENT_UNAVAILABLE_WITHIN_MS` under real load, that's this group's
+concurrency limit telling you it's undersized for your request volume: raise
+`numClients` for more concurrent capacity, or tune `maxWaitForClientMillis` to
+fail faster (surface backpressure to the caller sooner) or more patiently
+(absorb bursts, at the cost of tying up more request threads while waiting).
+
+=== It sizes each forked worker's view of available CPU
+
+Independently of the above, each group also auto-sizes its forked JVMs'
+`-XX:ActiveProcessorCount` from `numClients` and the host's core count — see
+xref:pipes/cpu-sizing.adoc[Forked-JVM CPU Sizing] for the full mechanics. This
+part *can* go wrong across groups: the auto-sizer for one group has no
+visibility into another group running in the same process, so if you enable
+both `/tika`/`/rmeta`/`/unpack` *and* `/pipes`/`/async` together — a config
+listing all of them, or simply leaving `endpoints` unset while
+`allowPipes=true` — each group's auto-sizer computes its slice as if it owned
+the whole host. Whether that actually causes oversubscription depends on your
+`numClients` values relative to the host's core count; it's not automatic, but
+it's also not something the auto-sizer will warn you about, because from
+either group's perspective alone the sizing looks fine. See
+xref:pipes/cpu-sizing.adoc#_known_limitation_multiple_pipes_groups_in_one_process[Known
+limitation: multiple Pipes groups in one process] for the mechanics and
+mitigation (scope `endpoints` to what you actually use, or set
+`-XX:ActiveProcessorCount` explicitly with the combined total in mind).
+
 == Topics
 
 * xref:using-tika/server/tls.adoc[TLS/SSL Configuration] — Secure your server with TLS and mutual authentication