HDFS-17872. Propagate tolerancePercentage to DataNode in DiskBalancer…#8335
Open
deepujain wants to merge 1 commit intoapache:trunkfrom
Open
HDFS-17872. Propagate tolerancePercentage to DataNode in DiskBalancer…#8335deepujain wants to merge 1 commit intoapache:trunkfrom
deepujain wants to merge 1 commit intoapache:trunkfrom
Conversation
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
The unit -1 is in TestBlockRecoveryCauseStandbyNameNodeCrash, which is unrelated to this DiskBalancer change. |
Contributor
Author
|
Rebased branch is already clean, and |
Contributor
Author
|
Pushed empty commit |
deepujain
force-pushed
the
HDFS-17872-diskbalancer-plan-tolerance-percent
branch
from
82ac70a to
f7b1341
Compare
Contributor
Author
|
Rebased this onto current apache/trunk, removed the old Trigger CI-only history, and force-pushed it back as a single clean JIRA commit. Local validation passed with JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home /opt/homebrew/bin/mvn -Dmaven.repo.local=/tmp/codex-m2 test -pl hadoop-hdfs-project/hadoop-hdfs -am -Dtest=TestNodePlan -DskipTests=false (5 tests, 0 failures). |
|
🎊 +1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dfs.disk.balancer.block.tolerance.percentis documented and used on the DataNode when executing a plan, but the plan command did not set tolerance percentage on plan steps. So the value sent to the DataNode was effectively the step default (0) unless the DataNode fell back to its own config. This change reads the config in the plan command and setstolerancePercenton each step so the plan propagated to the DataNode uses the same value.Change
setPlanParams(), readdfs.disk.balancer.block.tolerance.percentfrom configuration (default 10) and callstep.setTolerancePercent(tolerancePercent)for each step in each plan. So generated plans include tolerance and the DataNode receives it (it already usesstep.getTolerancePercent()when building work items).testPlanStepTolerancePercentInJson(): build a NodePlan with a MoveStep that hassetTolerancePercent(15), serialize to JSON, parse back, and assert the step’sgetTolerancePercent()is 15 (HDFS-17872).JIRA
Fixes HDFS-17872