[Client] Add client.lookup.max-inflight-requests-per-bucket to restrict lookup requests per server #4043

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.9.0 (latest release)

Please describe the bug 🐞

client.lookup.max-inflight-requests currently provides only a client-wide RPC concurrency limit.

During leader failover, such as a rolling upgrade, requests sent to an old or temporarily unavailable leader may remain incomplete until the connection fails or the request times out.

If many lookups target the same affected bucket, their RPCs may consume most or all global in-flight permits:

client.lookup.max-inflight-server (which is same as writer.max-inflight-requests-per-bucket

Once the global permits are exhausted, the lookup sender blocks before sending additional RPCs. Requests for healthy buckets and healthy TabletServers can no longer be dispatched, even though they are unrelated to the failed bucket.

To isolate this failure, the lookup client needs an additional per-bucket limit:

client.lookup.max-inflight-requests-per-bucket

Once one bucket reaches this limit, further requests for that bucket should remain queued, while requests for other buckets remain eligible for dispatch.

This is specifically per-bucket isolation. Multiple affected buckets on the same failed TabletServer can still collectively consume global permits; the option is not a per-node circuit breaker.

Solution

• client.lookup.queue-size : Maximum number of admitted logical lookup operations. |
• client.lookup.batch-timeout: Maximum wait time for an underfilled compatible batch.
• client.lookup.max-batch-size: Maximum number of logical lookups in each final RPC.
• client.lookup.max-inflight-requests : Maximum number of unacknowledged lookup RPCs across the client.
• client.lookup.max-inflight-requests-per-server: Maximum number of unacknowledged RPCs containing a given server.

Are you willing to submit a PR?

• I'm willing to submit a PR!