HBASE-29356 Incorrect split behavior when region information is missing #7035
+110
−11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR fixes the incorrect region split behavior when region size information is missing by skipping regions with unknown size during average size computation and preventing operations on them.
- Updated test mocks and added unit tests to verify that regions with unknown size are skipped.
- Modified the region size averaging logic in SimpleRegionNormalizer to account only for regions with known sizes.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java | Updated test mocks and added new test methods to handle regions with unknown sizes |
| hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java | Revised average region size calculation and added checks to skip regions with an unknown size |
Comments suppressed due to low confidence (1)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java:778
- [nitpick] The test method name 'testSplitOfLargeRegionIfOneIsNotKnow' is unclear; it likely should be 'testSplitOfLargeRegionIfOneIsNotKnown' for better readability.
@Test public void testSplitOfLargeRegionIfOneIsNotKnow() {
...e-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
Outdated
Show resolved
Hide resolved
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Ping @ndimiduk. Please take a look at this one? Seems reasonable. |
|
Heya @qqvpp thanks for the contribution. Can you please create for yourself a Jira account? We use Jira for project tracking. |
ndimiduk
reviewed
Jun 2, 2025
| if (targetRegionCount > 0) { | ||
| avgRegionSize = totalSizeMb / (double) targetRegionCount; | ||
| avgRegionSize = | ||
| totalSizeMb / (double) targetRegionCount - (regionCount - regionCountKnownSize); |
There was a problem hiding this comment.
If you're going to adjust the denominator here, I think that you also need to protect against a value <= 0. In that case, you can throw, like we do on entry into the method.
There was a problem hiding this comment.
I added a condition to verify the denominator and a test.
| } else { | ||
| avgRegionSize = totalSizeMb / (double) regionCount; | ||
| avgRegionSize = totalSizeMb / (double) regionCountKnownSize; |
There was a problem hiding this comment.
I think that you also need to protect against a 0 value here, in the same way.
There was a problem hiding this comment.
I added the condition, including the test modification.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We have identified a bug in the
SimpleRegionNormalizerlogic that leads to incorrect region splits when region size information is missing. If the size cannot be determined for one or more regions (e.g. due to unavailable metrics fromRegionServers), the average region size calculation becomes incorrect. This results in a scenario where all regions may be considered too large and get split unintentionally.Observed Behavior:
When region size data is not available (e.g.,
getRegionSizeMB()returns -1), the computed average size does not account for that, and regions with valid size may appear excessively large compared to the average — resulting in multiple unnecessary splits.Expected Behavior:
If region size is unknown for some regions, those regions should be skipped during normalization. The average region size should be computed only from the regions for which the size is known. No region should be split or merged unless its size is known.
Patch:
Skips regions with unknown size from average size computation.
Prevents split and merge operations on regions with unknown size.
Adds unit tests for scenarios with partial or total absence of size data.
Patch author: Milan Vymazal [email protected]
Tests:
testSplitOfLargeRegionIfOneIsNotKnowverifies correct behavior when one region has unknown size.testSplitOfAllUnknownSizeensures that no split happens if size data is missing for all regions.Reproduction:
Unfortunately, we are unable to reliably reproduce this bug in a live environment, since we cannot easily simulate the condition where RegionServer metrics are missing. However, we have confirmed the behavior through code analysis and the added unit tests.