-
Notifications
You must be signed in to change notification settings - Fork 3.3k
HBASE-29231 Throttles should support limits based on handler thread usage time #7000
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-29231 Throttles should support limits based on handler thread usage time #7000
Conversation
lucky number 7000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds thread handler usage throttling support to HBase by introducing a new throttle type (REQUEST_HANDLER_USAGE_MS) and integrating it with the quota infrastructure. Key changes include adding new parameters to quota‐checking methods, updating quota state and operation quota calculations, and extending proto definitions and client utilities to support the new throttle type.
Reviewed Changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
File | Description |
---|---|
TestThreadHandlerUsageQuota.java | Added tests to verify throttling for read and write operations using handler usage time. |
TestQuotaState.java & TestDefaultOperationQuota.java | Updated tests and quota constructors to include the new parameters. |
TestDefaultHandlerUsageQuota.java | Added tests for default handler usage quotas. |
TimeBasedLimiter.java | Introduced a new RateLimiter for request handler usage time and updated quota checking methods. |
RegionServerRpcQuotaManager.java | Integrated new requests-per-second supplier into quota instantiation. |
QuotaUtil.java, QuotaLimiter.java, NoopQuotaLimiter.java, GlobalQuotaSettingsImpl.java | Updated quota configuration and proto mappings to handle the new throttle type. |
ExceedOperationQuota.java & DefaultOperationQuota.java | Modified quota consumption logic to account for handler usage time and updated diff/consume methods. |
Protobuf files and client quota classes | Extended definitions and conversions to support REQUEST_HANDLER_USAGE_MS. |
Comments suppressed due to low confidence (1)
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/ExceedOperationQuota.java:104
- Passing 0L for the estimated handler usage time when grabbing quota for secondary limiters may be a placeholder; please verify if a more representative value should be propagated to better reflect actual usage.
limiter.grabQuota(numWrites, writeConsumed, numReads + numScans, readConsumed, writeCapacityUnitConsumed, writeCapacityUnitConsumed, isAtomic, 0L);
// low. | ||
return numHandlerThreads; | ||
} else { | ||
double requestsPerMillisecond = Math.ceil(requestsPerSecond / 1000); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nitpick] Consider adding documentation or comments to clarify the rationale for using Math.ceil(requestsPerSecond/1000) for estimating per-request handler usage time, ensuring it aligns with the intended behavior under low request rates.
Copilot uses AI. Check for mistakes.
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long actualSize, final long es | |||
private long calculateReadCapacityUnitDiff(final long actualSize, final long estimateSize) { | |||
return calculateReadCapacityUnit(actualSize) - calculateReadCapacityUnit(estimateSize); | |||
} | |||
|
|||
private long calculateHandlerUsageTimeEstimate(final double requestsPerSecond, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mind explain a bit more about the algorithm here? I do not fully understand what does 'handler usage time' mean...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this context, handler usage time
refers to the amount of time (in milliseconds) a handler thread will use.
We estimate the number of milliseconds we expect the thread to take to complete the request and deduct that amount from the quota. If there are enough milliseconds left in the quota, we proceed by taking that amount from the quota and continuing with the request. Otherwise, we throw a throttling exception.
When the request is complete, we calculate the actual amount of time used in the close()
method and adjust the quota by either adding back time or further subtracting it, as appropriate.
Does this make things clearer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not very familiar with the quota implementation, so for every operation, we will create a OperationQuota object and calculate the estimate time, and then compare it with the actual time usage, and see whether we should stop it?
In addition to the tests in this PR, I have deployed this to a test cluster and ran a load test against it. Everything is functioning as expected |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me — this should be a solid last line of defense for us when designing default throttles that will keep a wide variety of users in check
…sage time (#7000) Co-authored-by: Alex Hughes <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>
…sage time (#7000) Co-authored-by: Alex Hughes <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>
…sage time (#7000) Co-authored-by: Alex Hughes <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>
…n handler thread usage time (apache#7000) (will be in 2.6.3) Co-authored-by: Alex Hughes <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>
…sage time (#7000) (#7064) Signed-off-by: Ray Mattingly <[email protected]> Co-authored-by: Alex Hughes <[email protected]> Co-authored-by: Alex Hughes <[email protected]>
Upstream Issue
This PR implements thread handler usage throttling support in HBase, enabling administrators to limit the amount of thread handler time that can be consumed across all threads.
Features
REQUEST_HANDLER_USAGE_MS
that limits thread handler time usagehbase.quota.default.user.machine.request_handler_usage_ms
Implementation Details
TimeBasedLimiter
to track and limit thread handler usage timeDefaultOperationQuota
This throttling capability helps prevent individual users or applications from monopolising RegionServer handler threads, improving overall service stability and responsiveness for HBase deployments.