Skip to content

HBASE-28962 Meta replication is inconsistent after startup when reusi… #7046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: branch-2.6
Choose a base branch
from

Conversation

richardantal
Copy link
Contributor

…ng hbase storage location (#6448)

Copy link
Contributor

@taklwu taklwu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, other than the unit test result the patch is the same as #6448

@Apache-HBase

This comment has been minimized.

…ng hbase storage location (apache#6448)

Signed-off-by: Andor Molnár <[email protected]>
Signed-off-by: Wellington Chevreuil <[email protected]>
Reviewed-by: Aman Poonia <[email protected]>
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
_ branch-2.6 Compile Tests _
+1 💚 mvninstall 3m 27s branch-2.6 passed
+1 💚 compile 2m 58s branch-2.6 passed
+1 💚 checkstyle 0m 38s branch-2.6 passed
+1 💚 spotbugs 1m 38s branch-2.6 passed
+1 💚 spotless 0m 49s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚 mvninstall 3m 6s the patch passed
+1 💚 compile 2m 55s the patch passed
+1 💚 javac 2m 55s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 37s the patch passed
+1 💚 spotbugs 1m 43s the patch passed
+1 💚 hadoopcheck 17m 16s Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
+1 💚 spotless 0m 44s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 10s The patch does not generate ASF License warnings.
38m 59s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #7046
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux aef2cab960dd 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 31d7b6b
Default Java Eclipse Adoptium-11.0.23+9
Max. process+thread count 78 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 46s Docker mode activated.
-0 ⚠️ yetus 0m 5s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+1 💚 mvninstall 3m 20s branch-2.6 passed
+1 💚 compile 0m 50s branch-2.6 passed
+1 💚 javadoc 0m 26s branch-2.6 passed
+1 💚 shadedjars 6m 36s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 3m 11s the patch passed
+1 💚 compile 0m 50s the patch passed
+1 💚 javac 0m 50s the patch passed
+1 💚 javadoc 0m 26s the patch passed
+1 💚 shadedjars 6m 36s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 209m 39s hbase-server in the patch passed.
236m 44s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #7046
Optional Tests javac javadoc unit compile shadedjars
uname Linux c4c6fcd6ce5b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 31d7b6b
Default Java Eclipse Adoptium-11.0.23+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/testReport/
Max. process+thread count 4499 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 55s Docker mode activated.
-0 ⚠️ yetus 0m 7s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+1 💚 mvninstall 4m 39s branch-2.6 passed
+1 💚 compile 1m 9s branch-2.6 passed
+1 💚 javadoc 0m 31s branch-2.6 passed
+1 💚 shadedjars 6m 36s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 3m 19s the patch passed
+1 💚 compile 1m 0s the patch passed
+1 💚 javac 1m 0s the patch passed
+1 💚 javadoc 0m 28s the patch passed
+1 💚 shadedjars 6m 36s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 215m 42s hbase-server in the patch passed.
245m 24s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #7046
Optional Tests javac javadoc unit compile shadedjars
uname Linux 5f6e7c505f0b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 31d7b6b
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/testReport/
Max. process+thread count 4419 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 15s Docker mode activated.
-0 ⚠️ yetus 0m 5s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+1 💚 mvninstall 3m 28s branch-2.6 passed
+1 💚 compile 0m 42s branch-2.6 passed
+1 💚 javadoc 0m 26s branch-2.6 passed
+1 💚 shadedjars 5m 35s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 2m 29s the patch passed
+1 💚 compile 0m 41s the patch passed
+1 💚 javac 0m 41s the patch passed
+1 💚 javadoc 0m 24s the patch passed
+1 💚 shadedjars 5m 28s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 225m 24s hbase-server in the patch passed.
250m 14s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #7046
Optional Tests javac javadoc unit compile shadedjars
uname Linux 67aebd5896f2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 31d7b6b
Default Java Temurin-1.8.0_412-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/testReport/
Max. process+thread count 4189 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7046/3/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@Apache9 Apache9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not merge this.

Checked the code on branch-2.6, we also load the meta replica locations from master local region, not zookeeper. So the comment here is incorrect

#6448 (comment)

Please also revert the changes from existing branches, and let's review what is the root cause of this problem.

Thanks.

@richardantal
Copy link
Contributor Author

We run into this issue from time to time. When creating a cluster with storage location set to a storage location of an earlier cluster. Both clusters are set to use meta replication.
Only the default replica gets assigned on the new cluster.

Do you think this change could cause any problems?

@Apache9
Copy link
Contributor

Apache9 commented Jun 2, 2025

We run into this issue from time to time. When creating a cluster with storage location set to a storage location of an earlier cluster. Both clusters are set to use meta replication. Only the default replica gets assigned on the new cluster.

Do you think this change could cause any problems?

I agree that before identifying the root cause, you can apply some temporary approaches on your internal branches to 'fix' the problem.

But on the open source branch, I think we should find the root cause and then apply the 'correct' fix.

As I've said above, please check whether we have the master local region related changes in your internal branch.

And on the unit testing, please try delete everything on zookeeper to simulate your scenario. You can see HBASE-29292 and TestRecreateCluster on how to do this. If you can still produce the problem, we can check again whether we need the changes here.

Thanks.

@wchevreuil
Copy link
Contributor

Please do not merge this.

Checked the code on branch-2.6, we also load the meta replica locations from master local region, not zookeeper. So the comment here is incorrect

#6448 (comment)

Please also revert the changes from existing branches, and let's review what is the root cause of this problem.

Thanks.

@Apache9 is right, since HBASE-26193, present since 2.5.0, we don't rely on ZK for the region states anymore. @richardantal , have you still managed to reproduce this issue on a 2.5/2.6 based deployment?

wchevreuil added a commit to wchevreuil/hbase that referenced this pull request Jun 2, 2025
… startup when reusing hbase storage location (apache#6448)"

As discussed further on apache#7046, the described issue may not apply to branches containing
the changes from HBASE-26193 (present since 2.5.0).

This reverts commit 08621f9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants