HBASE-29231 Throttles should support limits based on handler thread usage time #7000
Conversation
|
lucky number 7000 |
There was a problem hiding this comment.
Pull Request Overview
This PR adds thread handler usage throttling support to HBase by introducing a new throttle type (REQUEST_HANDLER_USAGE_MS) and integrating it with the quota infrastructure. Key changes include adding new parameters to quota‐checking methods, updating quota state and operation quota calculations, and extending proto definitions and client utilities to support the new throttle type.
Reviewed Changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| TestThreadHandlerUsageQuota.java | Added tests to verify throttling for read and write operations using handler usage time. |
| TestQuotaState.java & TestDefaultOperationQuota.java | Updated tests and quota constructors to include the new parameters. |
| TestDefaultHandlerUsageQuota.java | Added tests for default handler usage quotas. |
| TimeBasedLimiter.java | Introduced a new RateLimiter for request handler usage time and updated quota checking methods. |
| RegionServerRpcQuotaManager.java | Integrated new requests-per-second supplier into quota instantiation. |
| QuotaUtil.java, QuotaLimiter.java, NoopQuotaLimiter.java, GlobalQuotaSettingsImpl.java | Updated quota configuration and proto mappings to handle the new throttle type. |
| ExceedOperationQuota.java & DefaultOperationQuota.java | Modified quota consumption logic to account for handler usage time and updated diff/consume methods. |
| Protobuf files and client quota classes | Extended definitions and conversions to support REQUEST_HANDLER_USAGE_MS. |
Comments suppressed due to low confidence (1)
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/ExceedOperationQuota.java:104
- Passing 0L for the estimated handler usage time when grabbing quota for secondary limiters may be a placeholder; please verify if a more representative value should be propagated to better reflect actual usage.
limiter.grabQuota(numWrites, writeConsumed, numReads + numScans, readConsumed, writeCapacityUnitConsumed, writeCapacityUnitConsumed, isAtomic, 0L);
| // low. | ||
| return numHandlerThreads; | ||
| } else { | ||
| double requestsPerMillisecond = Math.ceil(requestsPerSecond / 1000); |
There was a problem hiding this comment.
[nitpick] Consider adding documentation or comments to clarify the rationale for using Math.ceil(requestsPerSecond/1000) for estimating per-request handler usage time, ensuring it aligns with the intended behavior under low request rates.
Copilot uses AI. Check for mistakes.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
| @@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long actualSize, final long es | |||
| private long calculateReadCapacityUnitDiff(final long actualSize, final long estimateSize) { | |||
| return calculateReadCapacityUnit(actualSize) - calculateReadCapacityUnit(estimateSize); | |||
| } | |||
|
|
|||
| private long calculateHandlerUsageTimeEstimate(final double requestsPerSecond, | |||
There was a problem hiding this comment.
Mind explain a bit more about the algorithm here? I do not fully understand what does 'handler usage time' mean...
There was a problem hiding this comment.
In this context, handler usage time refers to the amount of time (in milliseconds) a handler thread will use.
We estimate the number of milliseconds we expect the thread to take to complete the request and deduct that amount from the quota. If there are enough milliseconds left in the quota, we proceed by taking that amount from the quota and continuing with the request. Otherwise, we throw a throttling exception.
When the request is complete, we calculate the actual amount of time used in the close() method and adjust the quota by either adding back time or further subtracting it, as appropriate.
Does this make things clearer?
There was a problem hiding this comment.
I'm not very familiar with the quota implementation, so for every operation, we will create a OperationQuota object and calculate the estimate time, and then compare it with the actual time usage, and see whether we should stop it?
|
In addition to the tests in this PR, I have deployed this to a test cluster and ran a load test against it. Everything is functioning as expected |
rmdmattingly
changed the title
…sage time (#7000) Co-authored-by: Alex Hughes <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>
…sage time (#7000) Co-authored-by: Alex Hughes <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>
…sage time (#7000) Co-authored-by: Alex Hughes <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>
…n handler thread usage time (apache#7000) (will be in 2.6.3) Co-authored-by: Alex Hughes <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>
…sage time (#7000) (#7064) Signed-off-by: Ray Mattingly <[email protected]> Co-authored-by: Alex Hughes <[email protected]> Co-authored-by: Alex Hughes <[email protected]>
Upstream Issue
This PR implements thread handler usage throttling support in HBase, enabling administrators to limit the amount of thread handler time that can be consumed across all threads.
Features
REQUEST_HANDLER_USAGE_MSthat limits thread handler time usagehbase.quota.default.user.machine.request_handler_usage_msImplementation Details
TimeBasedLimiterto track and limit thread handler usage timeDefaultOperationQuotaThis throttling capability helps prevent individual users or applications from monopolising RegionServer handler threads, improving overall service stability and responsiveness for HBase deployments.