[FEATURE] Enforce request `timeout` as a processing deadline (cancel/evict expired write requests)

[sean-]

Is your feature request related to a problem? Please describe

The ?timeout parameter on write APIs (_bulk, _index, _update, _delete) does not behave like a request deadline. It is used only to bound ClusterStateObserver waits on cluster-coordination conditions, and is never checked against elapsed processing or queue time. Concretely, in the current code request.timeout() is consumed in exactly three places, all retry paths driven by cluster-state changes:

• Coordinator: TransportBulkAction.BulkOperation builds an observer with bulkRequest.timeout() and only consults it when a global WRITE cluster block is present (handleBlockExceptions -> retry -> observer.isTimedOut()). The normal doRun() path performs no timeout check. (server/.../action/bulk/TransportBulkAction.java:604, :851-889)
• Shard primary: TransportShardBulkAction uses it to bound waiting on a dynamic mapping update. (.../action/bulk/TransportShardBulkAction.java:426)
• Replication reroute: TransportReplicationAction.ReroutePhase uses it to bound the wait-for-active-shards retry (UnavailableShardsException). (.../action/support/replication/TransportReplicationAction.java:978, :1192-1218, :1259-1261)

Implications:

1. Queued requests are never discarded when expired. Write-threadpool tasks execute in FIFO order regardless of how long they have waited; the only shedding mechanism is queue-capacity rejection (queue_size), not age/deadline.
2. No deadline during execution. Once a request begins executing, it runs to completion even if it has already far exceeded timeout. The startTime captured in doInternalExecute feeds only the response took field; it is never compared to timeout.

This makes client-side timeout + retry actively harmful under load. When write nodes are saturated:

• The client times out (e.g. at 2 min) and retries (e.g. via a Kafka consumer).
• The original request stays queued and will still execute.
• The retry enqueues duplicate work.
• Both execute, resulting in duplicated work that amplifies load instead of shedding it (a thundering-herd / retry-storm amplifier).

Describe the solution you'd like

Treat timeout as a true end-to-end deadline for write requests:

1. Capture an absolute deadline at request arrival (ingress / start of coordination) and propagate it through coordinator -> primary -> replication.
2. Check the deadline before dequeue execution. When a write-pool task reaches the head of the queue past its deadline, fail it fast with a timeout/429-style error instead of executing it, ideally evicting it from the queue rather than running then discarding.
3. Check the deadline at execution boundaries (e.g. before submitting shard sub-requests, between bulk items) so already-expired work fails fast.
4. Keep the existing cluster-coordination semantics; this adds processing-time enforcement rather than replacing the observer behavior.

Goal: an expired request becomes a cheap, fast failure (letting clients treat timeout as a non-retryable terminal error), instead of guaranteed-to-run work that compounds overload.

Related component

Indexing

Describe alternatives you've considered

• Client-side mitigation only (what we'd otherwise do): set very large client timeouts and mark client timeouts as non-retryable to avoid retry storms. This trades unbounded client latency for reduced amplification and does nothing for work already queued server-side.
• Rely on threadpool queue_size rejection / admission control (cf. #8910 Admission Controller framework). This sheds load by queue depth, not by per-request deadline, so it cannot drop a request that has already waited past its timeout once it's accepted.

Additional context

Documentation describes timeout narrowly (wait for primary availability / operations like automatic index creation and mapping updates), which is consistent with the code, but the broadly-held operator expectation is that ?timeout bounds total request time and lets the server abandon expired work. This is a request to make the parameter behave like a deadline.

[rmoles-cs]

+1 on this. Ran into this issue. Currently there is no way to set a timeout that will be respected by bulk indexer requests, which means if you have retry scenarios based on a timeout, you are simply adding duplicate load to the cluster and likely making your queuing situation worse!!